Design & Implement Real-Time Stream Processing Topologies

This skill designs scalable, fault-tolerant real-time stream processing topologies, with deep coverage of Kafka Streams and background knowledge of Apache Storm, Flink, and Pulsar as alternative frameworks. It distinguishes logical topology (the flow of data transformations and business logic) from physical topology (how those operations map to compute resources and partitions), recommending logical design first, then physical optimization for data locality and network overhead. Stream processing fundamentals cover treating events as immutable and append-only, correctly handling event time versus processing time, implementing exactly-once semantics through idempotent operations and transactional guarantees, and designing for graceful degradation under backpressure.

It provides working Kafka Streams patterns: a source-processor-sink topology (consuming an order stream, enriching it with a customer-table join, filtering by amount, and producing to an output topic), branching and fan-out (splitting a stream into premium, high-value, and standard branches by business logic, each routed to its own topic), and tumbling-window aggregation (summing sales by region over a 5-minute window advancing every minute). State management covers the distinction between stateless transformations and stateful aggregations backed by a state store, plus RocksDB state-store tuning (commit interval, cache buffering). Error handling covers custom deserialization and production exception handlers, and a dead-letter-queue pattern using a custom processor that forwards failed records to a separate topic.

Performance guidance covers thread and partition scaling configuration, replication and in-sync-replica settings, and buffering/caching/poll tuning. Monitoring tracks five key metrics - throughput, end-to-end latency, consumer lag, state-store size and growth, and error/DLQ rates - via JMX. Best practices cover keeping transformations simple and testable, minimizing state for scalability, planning for schema evolution with Avro or Protobuf, designing for idempotent reprocessing and replay, and unit-testing topologies with TopologyTestDriver.

// Kafka Streams topology example
StreamsBuilder builder = new StreamsBuilder();

// Source
KStream<String, OrderEvent> orders = builder.stream("orders", 
    Consumed.with(Serdes.String(), orderSerde));

// Processors
KStream<String, EnrichedOrder> enriched = orders
    .selectKey((key, order) -> order.getCustomerId())
    .leftJoin(customerTable, 
        (order, customer) -> enrichOrder(order, customer),
        Joined.with(Serdes.String(), orderSerde, customerSerde))
    .filter((key, enrichedOrder) -> enrichedOrder.getAmount() > 100.0);

// Sink
enriched.to("enriched-orders", Produced.with(Serdes.String(), enrichedOrderSerde));

Use this skill when designing or optimizing a real-time stream processing topology - building source-processor-sink pipelines, branching streams by business logic, windowed aggregation, state store tuning, or dead-letter-queue error handling.

It is not a fit as a deep reference for Storm, Flink, or Pulsar specifically - those are named as alternative frameworks but the concrete code patterns and tuning guidance here are Kafka Streams-specific.

Inputs are your event streams (topics) and the business logic for transforming, joining, and aggregating them. Outputs are working Kafka Streams topology code (source-processor-sink, branching, windowed aggregation), state store configuration, exception handling and DLQ routing, scaling configuration, and unit tests via TopologyTestDriver.

Backend and data engineers building real-time processing pipelines on Kafka Streams who need concrete, production-ready topology patterns, state management, and error handling - plus deployment guidance (container orchestration for scaling and failover, graceful shutdown, multi-region disaster recovery) for running topologies reliably in production.