4. Kafka Topic & Partition - stm29/Kafka GitHub Wiki

Kafka Topics

  • Kafka Will have default topic called __consumer_offsets
  • Topic is Kafka component where producers are connected
  • Publisher Publish a message in Kafka topic
  • Kafka in Topic is multi subscriber i.e., Topic can have more than 1 consumers
  • In Kafka Topic are logical entity, which means, when we say message is published to Topic, it means message published to PARTITIONS inside Topic

Kafka Topic Partition

  • Topics is divided into multiple parts called PARTITIONS
  • PARTITIONS can be considered as a linear Data structure same as ARRAY
  • Message are actually stored in Topic Patition
  • Every partition will have partition number
  • Every partition has increasing index (similar to array index where it starts from 0 and goes on) called OFFSET
  • New messages are always pushed at rear end
  • Data which published is immutable (can't be changed) after publish
  • We can have 1 r many partition in a single topic
  • In Multi Broker Kafka Cluster , Partitions are distributed across whole cluster i.e., For Ex: P0(Partition) will be on B1(Broker1), P2 will be in B2, P1 & P3 will be in B#

Message Publishing

  • 1Topic & 1Partition
    • Message will published
  • 1Topic & 2Partition
    • Message will get randomly published by default ,without any pattern followed ( But we can control it i.e., we can send a message to only one Partition )
  • 1Topic & 2Partition - redirecting message to particular partition
    • Partitioner method should contain following arguments
      • key_bytes
      • all_partition
      • available_partition