It is open-source lightweight, easy-to-manage message broker for general-purpose messaging or microservices integration. It is not only Point-to-Point but also supports Pub/Sub pattern.
Supports Advanced Message Queuing Protocol (AMQP) by default, along with STOMP and MQTT.
Used specially in moderate throughput is needed, no long-term message retention required.
Use-cases:
Asynchronous Messaging
Load Balancing/Parallel processing -distributes the tasks among consumers
Task Distribution- Distribute heavy background processes among workers
What are different components of RabbitMQ?
Producer: An application that sends messages to RabbitMQ.
Consumer: An application that receives messages from RabbitMQ.
Broker: The central RabbitMQ server that manages message routing, storage, and delivery.
Exchange: Receives messages from producers and routes them to queues based on rules (bindings and routing keys).
Queue: Buffers messages until they are consumed.
Binding: The relationship between an exchange and a queue, defining how messages should be routed.
Channel: A lightweight, virtual connection within a TCP connection, used for publishing and consuming messages.
Virtual Host (vhost): Provides logical separation for multi-tenant environments, allowing isolation of users, queues, and exchanges within the same broker instance.
Each RabbitMQ broker can host multiple virtual hosts (vhosts), and each vhost can contain its own set of exchanges, queues, bindings, and user permissions. e.g. There may be separate vhost for Dev, SIT, UAT on same cluster and each client must know the which vhost they are connecting to.
How does RabbitMQ works?
There can be one or more producers and consumers.
Fan-out is possible by copying same data to multiple queues.
Producers send data to exchange, that routes message to one or more appropriate queues.
Messages from one queue can be sent to one or more consumers in round-robin fashion. Broker will wait for consumer to finish processing before next message is sent.
If any time consumer goes down in middle, same message will be processed agin through another consumer.
What is RabbitMQ cluster and how does that works?
RabbitMQ cluster can be formed by deploying it to multiple nodes.
Each node is treated as one broker.
Each broker can have muliple vhosts(virtual hosts) for logically separating the queues, exchanges, users etc. (Separate vhosts for dev,uat etc.).
All nodes in the cluster share metadata such as users, virtual hosts, exchanges, and bindings.
RabbitMQ is built on Erlang distributed runtime environment, which manages the cluster. No External Controller/Configuration Service like zookeeper is needed.
Each queue has a single leader (primary replica) and one or more followers (replicas) distributed across cluster nodes. All operations (publishing, consuming, acknowledging) for a queue are handled by its leader. (Load Balancing)
Communications happens pretty much like Kafka from leader node through exchange for pushing data to any queue in cluster, queue replication is handled by broker.
If the leader node fails, RabbitMQ uses the Raft consensus protocol is used to elect a new leader from the followers.
How RabbitMQ is different from Kafka?
Kafka is used for high-throughput(millions of writes per sec) while RabbitMQ is for low or moderate load.
Kafka retains the messages for configured time, but RabbitMQ doesn't.
Kafka is pull based but RabbitMQ is push. RabbitMQ pushes data to consumers. Consumers can pull too but not recommended.
Kafka only supports TCP while RabbitMQ supports multiple protocols.
Kafka is good for streaming, real-time analytics, log aggregation etc, while RabbitMQ is for transactional systems, job queues, and microservices.
What is a Stream in RabbitMQ?
This feature introduced to provide high-throughput, log-based, persistent messaging similar to Apache Kafka, but overall doesn't work at same scale as Kafka.
Stream is like Topic and Partitioning can be does by creating grouping streams under super stream.