Producer Message Order - srivalligade04/ConfluentExamPreparationNotes GitHub Wiki

To preserve message, send order in Kafka, especially from a producer to a topic, you need to understand how Kafka handles partitioning and message sequencing.

Kafka Guarantees Message Order Per Partition

Kafka preserves the order of messages within a single partition. So, if you want to maintain the order of messages

1. Use a Single Partition (or Keyed Messages)

If all messages go to one partition, Kafka guarantees they will be written and read in order. If you use keys, Kafka will hash the key to determine the partition. All messages with the same key go to the same partition, preserving order per key.

2. Use One Producer Thread per Partition

If multiple threads or producers send to the same partition, ordering can be disrupted. Use a single-threaded producer or ensure synchronized access to the producer instance.

3. Set max.in.flight.requests.per.connection = 1

This ensures that only one message is in-flight at a time per connection. Prevents out-of-order delivery when retries happen.

4. Enable Idempotence

enable.idempotence=true

• Ensures exactly-once delivery and preserves order even during retries.
• Automatically sets:

1. acks=all
2. retries=Integer.MAX_VALUE

• max.in.flight.requests.per.connection=5 (can be overridden to 1 for strict ordering)

Summary Configuration for Strict Ordering:

• acks=all
• enable.idempotence=true
• max.in.flight.requests.per.connection=1
• retries=5
• key=some-consistent-key