RabbitMQ Most popular implementation of AMQP - tarunchhabra/parakalo GitHub Wiki
What is RabbitMQ?
RabbitMq- is a message-queueing software called message broker or queue manager (an intermediary for messaging).
RabbitMQ is a distributed message queue system. Distributed because it is usually run as a cluster of nodes where queues are spread across the nodes and optionally replicated for fault tolerance and high availability. (https://jack-vanlightly.com/blog/2017/12/4/rabbitmq-vs-kafka-part-1-messaging-topologies) AMQP is the core protocol for RabbitMQ (a Message Broker), but it also supports STORM, MQTT and HTTP through the use of plugins.
AMQP (Advanced Message Queuing Protocol)- a binary application layer protocol, designed to efficiently support a wide variety of messaging applications and communication patterns. It assists in the development of distributed, fault-tolerant and asynchronous applications. It provides wide range of features related to messaging, including-
Reliable queuing topic-based pub-sub messaging flexible routing transactions security
STOMP — a simple text based messaging protocol
MQTT — is a binary protocol known for its lightweight messaging
HTTP — you’re probably familiar with this one. It is not a messaging protocol, but management plugins in RabbitMQ use HTTP to send and receive messages.
-
RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.
-
It assists in the development of distributed, fault-tolerant and asynchronous applications.
RabbitMQ implement this characteristics:
**Security **—support authentication, authorization, LDAP, and TLS via RabbitMQ plugins.
Persistence- Messages are persisted to database abd able to survive broker crashes. First, some background: both persistent and transient messages can be written to disk. Persistent messages will be written to disk as soon as they reach the queue, while transient messages will be written to disk only so that they can be evicted from memory while under memory pressure. Persistent messages are also kept in memory when possible and only evicted from memory under memory pressure.
https://www.rabbitmq.com/persistence-conf.html
Reliability — confirms the message was successfully delivered to the message broker and confirms that the message was successfully processed by the consumer. RabbitMQ also have a builtin clustering feature that results to high availability, and scalability. There’s also an option to make your data persistent so the message wont be lost in case the broker quits or crashes.
Durability - Durable queues are persisted to disk and thus survive broker restarts. Queues that are not durable are called transient. Not all scenarios and use cases mandate queues to be durable.
Durability of a queue does not make messages that are routed to that queue durable. If broker is taken down and then brought back up, durable queue will be re-declared during broker startup, however, only persistent messages will be recovered RabbitMQ is durable nut kafka is highly durable.
Interoperability — message is transfer as stream of bytes so any clients can operate on it. RabbitMQ supports a lot of client libraries and and dev tools, in different programming languages.
**Open and Standard **— aside from following the open standards of AMQP, RabbitMQ is open source and anyone can contribute to make it better.
Ordering- RabbitMQ preserves the delivery order of messages.
Delivery Guarantees- It offers at-most once and at-least once but not exactly-once(offered by Kafka).
Here’s how it works:
Producers send/publish the messages to the broker -> Consumers receive the messages from the broker. RabbitMQ acts a communication middleware between both producers and consumers even if they run on different machines.
While the producer is sending a message to the queue, it will not be sent directly, but sent using the exchange instead. **The exchanges agents that are responsible routing the messages to different queues. **
So that the message can be received from the producer to the exchange and then again forwarded to the queue. This is known as the ‘Publishing’ method.
Afterwards, the message will be picked up and consumed from the queue; this is called ‘Consuming’.
By having a more complex application we would have multiple queues. So the messages (exchange) will send it in the multiple queues.
Messages are routed through exchanges (message routing agents) before arriving at queues. Hence it gives more control and flexibility over message routing. The exchange takes different message attributes into account, such as routing key, header attributes,. depending on the exchange type.
Binding: link that is set between a queue and an exchange.
Routing Key: message attribute based on which routing will happen.
Steps: The producer publishes a message to the exchange and identifies the routing key.
The exchange receives the message and is now responsible for the routing of the message.
A binding has to be set up between the queue and the exchange. In this case, we have bindings to three different queues from the exchange. The exchange routes the message into the queues.
The messages stay in the queue until they are handled by a consumer.
The consumer handles the message.
Message flow in RabbitMQ
There are three AMQP entities in RabbitMQ:
- Exchange
- Binding
- Queues
Messages published by a publisher are first received by the Exchange in RabbitMQ, then Exchanges will distribute message copies to Queues. To send appropriate messages to the appropriate queues, rules called Bindings are used.
Since Queues are consumer facing, it’s crucial that the Exchanges route messages to the appropriate Queues, and Bindings play an important role in this.
Once the messages reach Queues, messages can be delivered to the appropriate consumer or consumers can fetch the messages from the Queues.
When publishing a message, a publisher can pass attributes along with the message. The attributes can be used by RabbitMQ and the consumer applications.
What happens when a message fails to deliver to a consumer?
This can occur due to a network or application failure. If either of these failures was to occur, our system could potentially lose the message forever.
To address this issue, AMQP has a delivery acknowledgement mechanism in place. So a message will not be completely removed from a Queue unless we send a positive acknowledgment from the consumer. In the case of a negative acknowledgment, the message can be re-sent to the consumer or it can be dropped depending on the configuration settings by the publisher when sending the message.
In **AMQP 0–9–1, the applications **has more control over the message it sends, the application can define the exchange type that has to receive the message, and the queue where the message has to be saved. It can also define the routing scheme which binds the Exchange with Queues and it can control the message fetching and distribution logic on the Queues.
https://www.rabbitmq.com/tutorials/amqp-concepts.html
**Sending messages to multiple queues, exchange is connected to the queues by the binding and the routing key. **
A Binding is a “link” that you set up to connect a queue to an exchange. The Routing key is a message attribute()specified in header.
The exchange might look at this key when deciding how to route the message to queues (depending on exchange type).
Types of Exchanges
Direct (message goes to the queue where binding key exactly matching with routing key)
Topic (messages are routed based on the pattern match between routing and binding key)
Fanout (messages are broadcasted to all the queues bounded with an exchange. Routing key is ignored)
Headers (similar to topic, it matches message attributes [key-value] pair. It supports x-match=all/any whereas ‘all’ matches entire message attributes, ‘any’ looks for at least one match)
So,
Direct The binding key must match the routing key exactly - no wildcard support.
Topic Same as Direct, but wildcards are allowed in the binding key. '#' matches zero or more dot-delimited words and '*' matches exactly one such word.
Fanout The routing and binding keys are ignored - all published messages go to all
Each message broker implements the message channel concept in a different way.
For P2P communication RabbitMQ uses - Exchange + queue For Pub-Sub RabbitMQ uses fanout exchange and a queue per consumer
Let’s Sum up the core concept!
Producers emits the messages to exchange.
Consumers receive messages from the queues.
Binding connects an exchange with a queue using a binding key.
Exchange compares routing key with binding key.
Messages distribution depends on exchange type.
Exchange type: fanout, direct, topic and header.
Why RabbitMQ ?
You might need messaging if … you need to scale.
You might need messaging if … you need to monitor data feeds.
You might need messaging if … you need a message delivered responsibly.
You might need messaging if … you need things done in order.
You might need messaging if … you are using the cloud.
-- for distributing load.
Usage:
RabbitMQ is a messaging broker –an intermediary for messaging. We use it to provide asynchronous worker queues that are clustered, mirrored and persistent
The Integration Tier also uses RabbitMQ internally to distribute load across the integration cluster.
RabbitMQ needs to be configured with at least 2 ‘persistent’ nodes to guarantee that no messages are lost in case of one node failing in the cluster, see http://www.rabbitmq.com/ clustering.html.
Why do we need to use RabbitMQ?
Decouple: What means by decoupling is separate the core components of the application. This is what any application that implement micro-services wanted. Because their application will be maintainable and improve it’s quality of Single Responsibility Principle.
Flexibility: Because the application has been decoupled, so the application will be flexible enough for being develop to the next phase. But no only that flexible, because if you are using RabbitMQ, you will be able to connect 2 different apps/service that written by different app, these application will talk to each other by the help of a “translator” which is MOM.
Another benefit of using RabbitMQ:
- Highly Available Queue
- Multi-Protocol
- Many Client
- Clustering
- Management UI
- Tracing (Using dashboard can trace support)
- Plugin System (Extend core broker functionality in a variety of ways)
Steps to follow when setting up a connection and publishing a message/consuming a message
RabbitMQ allows multiple connection channels inside a single TCP connection in order to remove the overhead of opening a large number of TCP connections to the message broker.
https://medium.com/faun/rabbitmq-connection-vs-channel-aed1188a7f3a
Ref:
https://www.cloudamqp.com/blog/2015-05-18-part1-rabbitmq-for-beginners-what-is-rabbitmq.html
https://medium.com/faun/rabbitmq-connection-vs-channel-aed1188a7f3a
https://medium.com/ryans-dev-notes/learning-rabbitmq-3f59d11f66b4
https://medium.com/swlh/a-quick-guide-to-understanding-rabbitmq-amqp-ba25fdfe421d
https://medium.com/faun/rabbitmq-an-introduction-b84370fcf31
https://medium.com/ryans-dev-notes/learning-rabbitmq-3f59d11f66b4