Information Flow in Kafka Event‐Driven Application - ajay3003/doc GitHub Wiki

Information Flow in Kafka Event-Driven Application

1. Configuration Setup

  • Source: The appsettings.json file contains the configuration details, such as the Kafka broker (BootstrapServers) and the topic to use (Topic).
  • Flow: At startup, the application reads these settings using IConfiguration and injects them into the application via Dependency Injection (DI).
  • Target: The KafkaSettings model, which is shared across producer and consumer services.

2. Message Production (Producer Service)

  • Trigger: The Worker background service periodically generates a message to send.
  • Flow:
    1. The producer service (KafkaProducerService) uses Kafka settings like BootstrapServers and Topic to connect to Kafka.
    2. The service creates a Kafka message containing the message payload (e.g., "Message at {DateTime.Now}").
    3. It sends this message to Kafka using the Produce method of the Kafka producer client.
  • Target: Kafka stores the message in the specified topic (e.g., test-topic).

3. Kafka Broker

  • Role: Kafka acts as the central message bus. It stores messages in the specified topic, ensuring they are available for consumption by other components.
  • Flow:
    • Kafka maintains the message in the topic until a consumer acknowledges it or it expires (based on Kafka’s configuration).

4. Message Consumption (Consumer Service)

  • Trigger: The Worker background service runs the consumer in a separate task to listen for incoming messages.
  • Flow:
    1. The consumer service (KafkaConsumerService) uses Kafka settings to subscribe to the topic.
    2. It continuously polls Kafka for new messages.
    3. When a new message arrives, the consumer service processes it (e.g., logs it to the console or triggers further processing).
  • Output: The received message is processed and displayed/logged.

Information Flow Summary

  1. Input Configuration: Settings like Kafka broker address and topic name are read from appsettings.json and injected into the producer and consumer.
  2. Producer Action: The producer generates and sends a message to Kafka.
  3. Kafka Action: Kafka receives the message and stores it in the topic.
  4. Consumer Action: The consumer retrieves the message from Kafka and processes it.

Example Flow

  1. Producer:

    • Sends a message: "Message at 2024-12-19 10:15:00"
    • Kafka stores this message in the test-topic.

  2. Kafka Broker:

    • Receives and queues the message in the test-topic.

  3. Consumer:

    • Retrieves the message: "Message at 2024-12-19 10:15:00".
    • Logs the message: Message received: Message at 2024-12-19 10:15:00.

Diagram of Information Flow

[Configuration (appsettings.json)]
       |
       v
[KafkaProducerService] --> [Kafka (Broker/Topic)] --> [KafkaConsumerService]
       |                                                  |
       v                                                  v
 [Generate Message]                              [Process/Log Message]

What is happening "under the hood"?

  • Producer: Creates a connection to Kafka using the BootstrapServers and serializes the message.
  • Kafka: Manages the topic as a queue and stores the serialized messages until consumed.
  • Consumer: Connects to Kafka using the same broker address and subscribes to the topic, deserializing and processing messages as they arrive.

This flow ensures a loosely coupled, scalable, and fault-tolerant communication system.