Messaging - tarunchhabra/parakalo GitHub Wiki
Whats is messaging?
Message queuing facilitates effective communication between applications by sending messages. It also serves as a temporary shelter for messages by storing them while the destination application is busy or not connected.
Applications can connect to each other, as components of a larger application, or to user devices and data.
Messaging is asynchronous, decoupling applications by separating sending and receiving data.
Messaging enables software applications to connect and scale.
You may be thinking of data delivery, non-blocking operations or push notifications. Or you want to use publish / subscribe, asynchronous processing, or work queues. All these are patterns, and they form part of messaging.
What is a message?
It is a data transporter between a sender application and a receiver application.
What kind of data does the message consist of?
It could be a signal to inform an application to start processing a task, or tell an application about the completion of a task by another application. The message could hold crucial information required by the application for its own processing.
Basic Architecture of a Message Queue
All messages are generated by a Producer Application and pushed into a Message Queue. This process is known as Enqueuing.
Pushed messages will stay in this queue until a Consumer Application connects and fetches this message. This process is known as Dequeuing.
Both the Enqueue and Dequeue process are carried out independently by the Producer and Consumer applications, and because of this independent nature of producer and consumer, we’ve got the option to hold a message in a message queue, waiting for a consumer to fetch it.
So this process of a message getting pushed to a message queue by a producer application and getting consumed by a consumer application is referred to as Message Queueing.
Message Brokers
Message brokers take care of the connections between applications. Based on the requirement, a bi-directional connection will be created between each application system and message broker. Messages are then transported via this connection. Message brokers act as a hub to route appropriate messages to appropriate destinations.
In a nutshell, a Message Broker routes appropriate messages to their appropriate destinations, similar to the way a Telephone Switching Office works.
Examples- Apache Kafka(Topic), Rabbit MQ(Exchange and Queues), AWS Kinesis(Streams), AWS SQS(Queue)- SQS only supports P2P
Advantages of Messaging:
-
Loose coupling between services.
-
Message Buffering
-
Flexible Communication
DisAdvantages of Messaging:
- Potential Single Point of failure.
When selecting a message broker, you have various factors to consider, including the following:
Supported programming languages— You probably should pick one that supports a variety of programming languages.
Supported messaging standards— Does the message broker support any standards, such as AMQP and STOMP, or is it proprietary?
Messaging ordering— Does the message broker preserve ordering of messages?
Delivery guarantees— What kind of delivery guarantees does the broker make?
Persistence— Are messages persisted to disk and able to survive broker crashes?
Durability— If a consumer reconnects to the message broker, will it receive the messages that were sent while it was disconnected?
Scalability— How scalable is the message broker?
Latency— What is the end-to-end latency?
Competing consumers— Does the message broker support competing consumers?
Each broker makes different trade-offs.
For example, a very low-latency broker(Kafka) might not preserve ordering, make no guarantees to deliver messages, and only store messages in memory.
A messaging broker that guarantees delivery and reliably(Rabbit MQ) stores messages on disk will probably have higher latency.
RabbitMQ vs Kafka
What is the differences between RabbitMQ and Kafka?
Kafka -> Dumb Broker, Smart Consumer to read its buffer
RabbitMQ -> Smart Broker, Dumb Consumer to read its buffer
Distributed RabbitMQ- is not decentralized. Kafka - distributed
Message Ordering Queues in RabbitMQ are ordered collections of messages. Messages are enqueued and dequeued (consumed) in the FIFO manner, although priority queues, sharded queues and other features may affect this. First of all, Kafka only guarantees message ordering within a partition, not across partitions. This places a burden on the producers and consumers to follow certain Kafka design patterns to ensure ordering. https://stackoverflow.com/questions/48448066/how-does-kafka-guarantee-message-ordering-as-processed-by-consumers-across-parti
Durability Durable queues are persisted to disk and thus survive broker restarts. Queues that are not durable are called transient. Not all scenarios and use cases mandate queues to be durable.
Durability of a queue does not make messages that are routed to that queue durable. If broker is taken down and then brought back up, durable queue will be re-declared during broker startup, however, only persistent messages will be recovered RabbitMQ is durable nut kafka is highly durable. Similarities: Kafka now has transactions support like RabbitMQ. Both of them are reliable/persistence means - messages are persisted to disk and able to survive broker crashes. Both have failover/High Availability
https://microservices.io/patterns/communication-style/messaging.html https://www.rabbitmq.com/queues.html
JMS vs RabbitMQ/Kafka
a) Message delivery failures Apache Kafka : client is responsible for replaying failed messages Traditional message brokers : broker is responsible for re-delivery of message
b) Scalability Apache Kafka : horizontally scalable by adding more partitions Traditional message brokers : not possible to scale horizontally c)Design Apache Kafka : client-centric Traditional message brokers : broker-centric https://manasvigupta.github.io/apache-kafka-vs-traditional-message-brokers/
Messaging
https://www.developertoarchitect.com/lessons/lesson22.html
Request-Reply Messaging communication
https://www.developertoarchitect.com/lessons/lesson1.html
Pub/Sub Messaging
https://www.developertoarchitect.com/lessons/lesson61.html
Data durability or Preventing Data Loss When Using Messaging
https://www.developertoarchitect.com/lessons/lesson70.html
Kafka vs RabbitMQ
https://www.developertoarchitect.com/lessons/lesson2.html