Messages in a batch (Kafka Producer Side) - srivalligade04/ConfluentExamPreparationNotes GitHub Wiki
Kafka producers batch messages together before sending them to the broker to improve throughput and network efficiency.
How Batching Works
Messages are accumulated in a batch per partition.
A batch is sent when
It reaches the batch.size (in bytes), or The linger.ms timeout expires (even if the batch isn't full).
Key Configs
- batch.size: Max size of a batch (default: 16 KB).
- linger.ms: Max time to wait before sending a batch (default: 0 ms = send immediately). Example If batch.size=32KB and linger.ms=10, the producer will wait up to 10 ms to fill the batch before sending it.
How the Broker Processes Batches
When a batch arrives at the broker:
The broker appends the batch to the partition log. It replicates the batch to follower replicas (if acks=all). Once replication is confirmed (based on acks), the broker sends an acknowledgment to the producer. Kafka brokers are optimized to handle batches efficiently, reducing disk I/O and improving throughput.
max.in.flight.requests.per.connection
This setting controls how many unacknowledged requests the producer can send in parallel to a broker.
Why It Matters:
If set > 1, the producer can send multiple batches without waiting for the previous one to be acknowledged. This improves throughput but can cause out-of-order delivery if retries happen.
For Ordering Guarantees: max.in.flight.requests.per.connection=1 This ensures when combined with enable.idempotence=true acks=all