Messages in a batch (Kafka Producer Side) - srivalligade04/ConfluentExamPreparationNotes GitHub Wiki

Kafka producers batch messages together before sending them to the broker to improve throughput and network efficiency.

How Batching Works

Messages are accumulated in a batch per partition.

A batch is sent when

It reaches the batch.size (in bytes), or The linger.ms timeout expires (even if the batch isn't full).

Key Configs

• batch.size: Max size of a batch (default: 16 KB).
• linger.ms: Max time to wait before sending a batch (default: 0 ms = send immediately). Example If batch.size=32KB and linger.ms=10, the producer will wait up to 10 ms to fill the batch before sending it.

How the Broker Processes Batches

When a batch arrives at the broker:

The broker appends the batch to the partition log. It replicates the batch to follower replicas (if acks=all). Once replication is confirmed (based on acks), the broker sends an acknowledgment to the producer. Kafka brokers are optimized to handle batches efficiently, reducing disk I/O and improving throughput.

max.in.flight.requests.per.connection

This setting controls how many unacknowledged requests the producer can send in parallel to a broker.

Why It Matters:

If set > 1, the producer can send multiple batches without waiting for the previous one to be acknowledged. This improves throughput but can cause out-of-order delivery if retries happen.

For Ordering Guarantees: max.in.flight.requests.per.connection=1 This ensures when combined with enable.idempotence=true acks=all