Durability ‐ Acknowledgement Settings - srivalligade04/ConfluentExamPreparationNotes GitHub Wiki

Kafka producers maintain durability through a combination of acknowledgment settings, retries, and replication. Here's how it works:

1. Acknowledgment (acks) Settings

This controls how many broker acknowledgments the producer waits for before considering a message as successfully sent:

• acks=0: No durability guarantee. The producer doesn't wait for any acknowledgment.
• acks=1: The leader broker writes the message to its local log and acknowledges. If the leader crashes before followers replicate, data may be lost.
• acks=all or acks=-1: The leader waits for all in-sync replicas (ISR) to acknowledge. This provides strong durability.

2. Retries and Backoff

retries: If a transient error occurs (e.g., network glitch), the producer retries sending the message. retries.backoff.ms: Wait time between retries to avoid overwhelming the broker. This ensures that temporary issues don’t result in message loss.

3. Replication

Kafka topics are typically configured with multiple replicas (e.g., 3). When a message is written:

It goes to the leader partition. The leader replicates it to follower replicas. If acks=all, the message is only acknowledged after all ISR replicas confirm receipt. This ensures that even if a broker fails, another replica can take over without data loss.

4. Idempotence and Exactly-Once Semantics (EOS)

enable.idempotence=true: Ensures that retries don’t result in duplicate messages. EOS: With additional settings (transactional.id, etc.), Kafka can guarantee exactly-once delivery even across retries and failures.

5. Delivery Timeout

delivery.timeout.ms: Maximum time the producer will try to send a message before giving up. It bounds the total time spent on retries.

Flow Summary:

• Producer sends message.
• Broker (leader) writes to log.
• If acks=all, waits for ISR replicas to confirm.
• If failure occurs, producer retries (if configured).
• Once acknowledged, the message is considered durable.