Build Robust Kafka Consumers

This skill builds robust Apache Kafka consumers with proper offset handling, error recovery, and performance tuning. Core configuration guidance covers using multiple bootstrap servers for availability, meaningful group.id naming, matching (de)serializers to the producer, disabling auto-commit for at-least-once guarantees, and choosing auto.offset.reset (earliest for reprocessing, latest for real-time-only). Performance settings cover fetch.min.bytes, fetch.max.wait.ms, max.poll.records, session.timeout.ms, and a heartbeat interval set to roughly a third of the session timeout.

A Java RobustKafkaConsumer polls in a loop, processes each record, and commits synchronously only after successful processing, catching WakeupException for graceful shutdown via a running flag and consumer.wakeup(). An advanced variant adds per-record retry with exponential backoff up to a max retry count, routing permanently failed records to a dead-letter topic via a separate producer, and tracking per-partition offsets in a map committed together after each poll batch.

while (!success && attempts < maxRetries) {
    try { processRecord(record); success = true; }
    catch (Exception e) { attempts++; Thread.sleep(1000 * attempts); }
}
if (!success) sendToDeadLetterQueue(record);

The equivalent Python implementation uses kafka-python's KafkaConsumer with the same manual-commit, retry-with-backoff, and dead-letter-queue pattern, JSON deserialization, and commits after processing regardless of success (since failures are already routed to the DLQ). Consumer group management best practices cover implementing a ConsumerRebalanceListener for graceful partition reassignment, using the cooperative-sticky assignment strategy to minimize rebalance disruption, and choosing an offset strategy (at-least-once via manual commit, at-most-once via pre-processing auto-commit, or exactly-once via transactional consumers). Monitoring covers consumer lag, processing rate, error/retry counts, and rebalance frequency, with a Java health-check class tracking time since last poll and exposing lag/rate metrics. Common pitfalls flagged include auto-commit with long processing times, ignoring rebalance events, and committing offsets for failed messages.

Use this skill when building or hardening a Kafka consumer that needs reliable offset management, retry logic with a dead-letter queue for failed messages, graceful rebalance handling, or health/lag monitoring. It applies to both new consumer implementations in Java or Python and adding resilience to an existing basic consumer.

It is not the right tool for simple fire-and-forget consumption where auto-commit and no retry logic is acceptable, or for exactly-once processing requirements without transactional producers/consumers configured - that needs Kafka's transactional API rather than the at-least-once patterns shown here.

Input: the Kafka topic(s) to consume, the processing logic to apply per record, and the required delivery guarantee (at-least-once, at-most-once, exactly-once). Output: a Java or Python consumer implementation with manual offset commits, retry-with-backoff and dead-letter-queue routing for failures, rebalance-safe partition handling, and health/lag monitoring metrics.

Built on the Apache Kafka Java client (KafkaConsumer, KafkaProducer, ConsumerRebalanceListener) and Python's kafka-python library, with dead-letter-queue routing via a secondary Kafka producer and metrics exposed through the consumer's native metrics() API.

Backend and data engineers building Kafka-based event processing pipelines - particularly those needing reliable at-least-once consumption with retry and dead-letter handling, and observability into consumer lag and rebalancing behavior.