Consumer offset settings - srivalligade04/ConfluentExamPreparationNotes GitHub Wiki

In Apache Kafka, consumer offset settings determine how and where a consumer starts reading messages from a topic. These settings are crucial for message processing reliability, restarts, and rebalancing. Here's a breakdown of the key settings:

Key Consumer Offset Settings

image

Common auto.offset.reset Values

  • latest (default): Start from the end of the log (new messages only).
  • earliest: Start from the beginning of the log (replay all messages).
  • none: Throw an error if no previous offset is found.

Best Practices

  • Set enable.auto.commit=false and manually commit offsets after processing to avoid data loss.
  • Use auto.offset.reset=earliest for reprocessing or new consumers.
  • Monitor and store offsets in an external store (e.g., database) for fine-grained control.