Process Real-Time Data Streams with Kafka

This skill provides expertise in Apache Kafka Streams for building robust, scalable real-time stream processing applications, covering the Streams DSL, Processor API, state management, windowing, and production deployment. It explains the stream-table duality (streams as an immutable append-only event sequence, tables as the mutable latest value per key, GlobalKTable replicated across all instances vs. KTable partitioned by key), exactly-once semantics (processing.guarantee=exactly_once_v2 for transactional processing, idempotent producers via enable.idempotence=true, and proper error handling/recovery), and state store management (RocksDB for large state vs. in-memory for speed, changelog topic configuration for fault tolerance, interactive queries for external access).

KStream<String, Transaction> transactions = builder.stream("transactions");

// Tumbling window aggregation
KTable<Windowed<String>, Double> hourlySpending = transactions
    .groupByKey()
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofHours(1)))
    .aggregate(
        () -> 0.0,
        (key, transaction, aggregate) -> aggregate + transaction.getAmount(),
        Materialized.<String, Double, WindowStore<Bytes, byte[]>>as("hourly-spending")
            .withValueSerde(Serdes.Double())
            .withRetention(Duration.ofDays(7))
    );

It covers stream-stream joins (windowed, e.g. joining orders and payments within a 10-minute window via JoinWindows.ofTimeDifferenceWithNoGrace), stream-table joins for enrichment (e.g. joining orders against a customer KTable), and advanced windowing: session windows for user activity (grouped by an inactivity gap, e.g. 30 minutes), and sliding windows for moving averages over a time difference. For error handling it covers custom deserialization/production exception handler classes and a custom setUncaughtExceptionHandler that can choose to replace the thread or shut down the application, plus interactive queries against a ReadOnlyWindowStore for external state access.

For production deployment it recommends setting topic partitions to match expected parallelism (num_stream_threads * num_instances), consistent partitioning across related topics, monitoring consumer lag, tracking key metrics (commit-latency, process-latency, record-cache-hit-ratio, state-store-record-count, thread-start-time), and a graceful shutdown hook that closes the Streams instance with a timeout. For performance it recommends minimizing repartitioning by grouping transformations, using selectKey() judiciously since it triggers repartitioning, leveraging co-partitioning for stream-stream joins, using GlobalKTable for small reference data, and tuning RocksDB state store settings (block cache size, write buffer size, max write buffers) with logging and caching enabled.

Use it when building or hardening a Kafka Streams application: windowed aggregations, stream-stream or stream-table joins, exactly-once processing guarantees, state store tuning, or production deployment concerns like partitioning, monitoring, and graceful shutdown.

It is scoped to Kafka Streams specifically (DSL and Processor API), not general Kafka producer/consumer usage or other stream processing frameworks.

Input is a stream processing requirement - an aggregation, join, windowing pattern, or production hardening need. Output is Java Kafka Streams code: topology definitions using the DSL (KStream/KTable/GlobalKTable), StreamsConfig properties for exactly-once and performance tuning, exception handlers, interactive query methods, and production deployment configuration (partitioning, monitoring metrics, graceful shutdown).

Backend and data engineers building real-time stream processing applications on Kafka Streams who need production-grade patterns for windowed aggregations, joins, exactly-once semantics, and operational hardening.