Mosquitto Overview - padogrid/padogrid GitHub Wiki

◀️ Mosquitto CLASSPATH :link: Installing Building Mosquitto ▶️

Mosquitto is a light-weight MQTT broker maintained by Eclipse. By design, it is made to run on small edge devices with limited resources. At startup, it takes ~830 KB of memory (PSS) for mqtt and ~2700 KB for websockets.

Keeping Mosquitto resource friendly comes with a few limitations, however.

It runs as a standalone server.
It is limited to one-way traffic to a single active backup broker with no support for client failover.
It has no support for high availability (HA).
It is single threaded.
It does not scale.

Standalone

Clearly, with these limitations, Mosquitto is not meant for enterprise applications, which must be resilient to failures and able to scale out as needed. To lift the limitations, we need a way to cluster Mosquitto brokers. By clustering, we can

Provide an HA service.
Scale out by simply launching more brokers.
Provide a load-balancing service to increase the overall performance.

More formally, the clustering solution must

Lift the standalone server limitations.
Run Mosquitto as is requiring no modifications.
Not include any external dependencies to Mosquitto.
Conform to all MQTT QoS levels.

Our solution is to cluster Mosquitto brokers from the client side.

Normally, clustering is done from the server side, but to keep Mosquitto brokers unchanged, our solution clusters them from the client side. We do this by introducing the concept of virtual clusters. A virtual cluster is a highly available and scalable collection of endpoint connections maintained by the client. Creating virtual clusters is possible for MQTT brokers since they do not store data, and for the most part, they are stateless. They merely pass data from publishers to subscribers, and data loss and consistency are handled by QoS. With a proper failover mechanism in place, if a broker fails, the virtual cluster seamlessly continues broker services uninterrupted.

Virtual Clsuter

QoS

QoS 0 - at most once delivery

This "fire-and-forget" service level guarantees a best-effort delivery. It does not guarantee the message will be delivered to the subscriber but it does guarantee that the subscriber will not receive duplicate messages.

Our solution cannot provide this service level since there is a possiblity of delivering duplicate messages. For example, if there is a network failure after the broker has delivered the message to the subscriber but before the client completes the transaction, then our solution resends the message using another broker, effectively deliverying a duplicate message. This is not a bad thing, however. Most applications prefer duplicate messages over lost messages. In fact, the higher level, QoS 1 delivers duplicate messages.

QoS 0.5 - quasi QoS 1

Because our solution cannot support QoS 0 but does better than QoS 0, we introduce QoS 0.5 to describe this discrepancy. QoS 0.5 is equivalent to QoS 1 if the broker fails but the failover broker is successful. On the other hand, it is equivalent to QoS 0 if there is only one broker remain in the virtual cluster and a failover cannot be achieved.

QoS 0.5 is only used to describe our solution. The virtual cluster understands only QoS 0, 1, and 2. If you configure it to QoS 0, then you will get the QoS 0.5 effect.

QoS 1 - at least once delivery

QoS 1 guarantees that a message is delivered at least once to the receiver. That is, a message can potentially be delivered multiple times.

Our solution is compliant with this service level since QoS 0.5 is the baseline.

QoS 2 - exactly once delivery

QoS 2 guarantees that each message is received only once by the receiver.

Our solution is compliant with this service level since it guarantees the message will not be delivered if there is a broker communication failure. We can safely resend the failed message to another broker with QoS 2 still intact.

Solution: `HaMqttClient`

HaMqttClient takes a novel approach of clustering servers from the client side. It includes support for clustering MQTT brokers (Mosquitto and others) by periodically probing and capturing network reachable brokers. It treats each broker as a standalone server and reels in if detected to its live broker list. For all client MQTT operations, it only uses the brokers in the live list. This means it performs load balancing and failover only on the live brokers for any client operations. This makes it fast and seamless.

For sending a message, it uses only one live broker. This guarantees that no duplicate messages will be published over multiple brokers.
For receiving messages, it subscribes to all the live brokers. The idea behind this is that the client will receive from the broker that the sender published the message to. This is possible since the client is subscribed to all the live brokers.
HaMqttClient supports multiple clusters. You can configure multiple clusters, each having their own set of brokers. Moreover, a broker can belong to multiple clusters.
HaMqttClient implements Paho's IMqttClient interface and all of the additional methods in the implementation class, MqttClient. This makes migrating Paho applications to HaMqttClient as simple as changing a single line of code.
HaMqttClient can dynamically scale in or out. You can remove or add brokers in each cluster at any time.
HaMqttClient can be configured via API or a configuration file. A configuration file is preferred over API. You can define one or more clusters identifiable by unique cluster names in the configuration file. From your code, you would use the cluster names to access the HaMqttClient instances.

`HaMqttClient` Tradeoffs

Multiple retained messages. Since HaMqttClient subscribes to multiple brokers, multiple retained messages will be delivered for a given topic. There are workarounds to this limitation. See HaMqttClient Limitations: Multiple Retained Messages.
Potential cluster configuration inconsistency. Since clusters must be configured by each client, a degree of discipline is required to prevent inconsistent configurations between clients. This can be easily handled by creating a cluster management tool that can synchronize client configurations.
MqttClient wraps Paho's MqttClient, which has some connectivity issues. For example, creating two MqttClient instances and invoking MqttClient.connect() causes the second instance to fail with a protocol error. HaMqttClient has a defensive mechanism to handle such an error. In worst scenario, if the error cannot be handled, then the broker will be removed from the live list to the dead list and a new connection will be made in the next probing cycle. The client may never experience this error since it is handled behind the scene, but there may be a brief period of time when load balancing maybe performed on less than full cluster capacity.