Consumer offset settings - srivalligade04/ConfluentExamPreparationNotes GitHub Wiki
In Apache Kafka, consumer offset settings determine how and where a consumer starts reading messages from a topic. These settings are crucial for message processing reliability, restarts, and rebalancing. Here's a breakdown of the key settings:
Key Consumer Offset Settings
Common auto.offset.reset Values
- latest (default): Start from the end of the log (new messages only).
- earliest: Start from the beginning of the log (replay all messages).
- none: Throw an error if no previous offset is found.
Best Practices
- Set enable.auto.commit=false and manually commit offsets after processing to avoid data loss.
- Use auto.offset.reset=earliest for reprocessing or new consumers.
- Monitor and store offsets in an external store (e.g., database) for fine-grained control.