Data Transport and Communication Patterns - kdwivedi1985/system-design GitHub Wiki

Synchronous- Request-Response/ One-to-One/ Point-to-Point/ (Client-Server):

  • One component(Client) sends a request and waits for a response.

Asynchronous Communication:

  • Message Queue, Pub/Sub, Even Streaming, Change Data Capture(CDC), Event Bus, Saga, Workflow orchestration, Notification systems are different applications or use cases of Asynchronous Communication.

  • One-to-One/ Point-to-Point/ Message Queuing - Client makes request but doesn't wait for response. (RabbitMQ, Kafka)

    • Producer sends data to a queue, and consumer reads data from queue in order it was received (FIFO). (Amazon SQS, RabbitMQ)

  • One-to-Many/ Many-to-Many/ Many-to-one:

    • Publish/Subscribe (Pub/Sub) - Use Kafka (One or more publisher and one subscriber) [m-1, 1-1]
    • Fanout - Use Kafka (Create multiple consumer groups on Kafka topic and each consumer can point to one of those to get complete data). [1-m, m-m]

  • Batch Processing:

    • Job Scheduling/ Workflow Orchestration - Use Custom logic or Snaplogic, Dell Boomi, mulesoft etc for workflow orchestration.

Different terminologies used by different integration platforms. Here are few common once:

  • Enterprise Service Bus:

    • ESB acts a bus, which transports data between system. It facilitates real-time data exchange between systems within org. It enables SOA (Service Oriented Architecture). Acts as bridge with different communication protocols eb. SOAP-to-REST etc. (ETL is for async transfer) Used in Mulesoft. It is central backbone for integration between systems.
    • Use case - If multiple communication protocols are used.

  • Message Broker:

    • A message broker is an architecture pattern facilitates communication and data exchange between different parts of a distributed systems or services. It acts as an intermediary, receives messages, validates, transforms, routes and delivers reliably between producer and one or more consumers asynchronously. Used in Kafka, RabbitMQ, ActiveMQ. It is like Hub-and-Spoke. Acts as central hub which routes messages between producers and consumers.
    • Use case- Simple decoupling, event distribution or queuing.