004. Kafka Theory. Producers and Message Keys - MarkHuntDev/my-kafka-exercises GitHub Wiki

Producers

  • Producers write data to topics (which is made of partitions)
  • Producers automatically know to which broker and partition to write to
  • In case of Broker failures, Producers will automatically recover
  • Producers can choose to receive acknowledgment of data writes:
    • acks=0: Producer won't wait for acknowledgement (possible data loss)
    • acks=1: Producer will wait for leader acknowledgement (limited data loss)
    • acks=all: Leader + replicas acknowledgment (no data loss)

Producers: Message keys

  • Producers can choose to send a key with the message (string, number, etc...)
  • If key=null, data is sent round robin (broker 101 then 102 then 103...)
  • If a key is sent, then all messages for that key will always go to the same partition
  • A key is basically sent if you need message ordering for a specific field (ex: truck_id)