Consumer offset lag - srivalligade04/ConfluentExamPreparationNotes GitHub Wiki

In Apache Kafka, consumer offset lag is a key metric that shows how far behind a consumer is in processing messages from a topic partition.

What Is Consumer Offset Lag?

Offset lag is calculated as:

  • Offset Lag = Latest Offset in Partition - Last Committed Offset by Consumer
  • Latest Offset: The most recent message offset in the partition.
  • Committed Offset: The last offset the consumer has acknowledged as processed.

Why It Matters

High lag means the consumer is falling behind, possibly due to:

  • Slow processing
  • Network issues
  • Not enough consumers
  • Zero or low lag means the consumer is keeping up with the producer.

How to Monitor Offset Lag

Kafka Monitoring Tools:

  • Kafka Manager, Confluent Control Center, Burrow, Cruise Control
  • These tools visualize lag per consumer group and partition.
  • Command Line: kafka-consumer-groups.sh --bootstrap-server --describe --group <group_id>

Output includes:

  • Topic
  • Partition
  • Current Offset
  • Log End Offset
  • Lag

JMX Metrics:

Kafka exposes lag metrics via JMX (Java Management Extensions) for integration with Prometheus, Grafana, etc.

Best Practices

  • Set up alerts for high lag thresholds.
  • Use consumer groups wisely to parallelize consumption.
  • Tune max.poll.records, fetch.min.bytes, and session.timeout.ms for performance.
⚠️ **GitHub.com Fallback** ⚠️