Cluster Archetypes - padogrid/padogrid GitHub Wiki

◀️ Clustering MQTT :link: Enabling Mosquitto SSL/TLS ▶️

To simplify configuring virtual clusters, we identify common use cases in terms of eight (8) archetypes. Each archetype represents a messaging architecture that applications can select and apply based on their requirements.

Each archetype defined in this section is presented in the bundle, Mosquitto/MQTT Virtual Cluster Tutorial.

Archetype 1

Archetype 1 refers to a baseline virtual cluster providing the effect of a fan-out architecture: one publisher and many subscribers. In Archetype 1, the receiver subscribes to all the brokers in the virtual cluster. The sender delivers each message using a single publisher that is connected to one of the brokers. The receiver receives the message via the subscriber connected to that broker. The publisher is selected by one of the methods shown below. The default method is publisherType: STICKY. The publisher "sticks" to one endpoint for all operations.

  • By publisherType. Valid values are STICKY, RANDOM, ROUND_ROBIN, and ALL.
  • By endpoint name. Each endpoint is identifiable by a unqiue name.
  • By topic base. Each endpoint can be tied to a topic base.
clusters:
  - name: edge
    connection:
      serverURIs: [tcp://localhost:1883-1892]

Archetype 1

Archetype 2

Archetype 2 is inverse of Archetype 1. Instead of targeting individual brokers, it targets all brokers in the virutal cluster providing the effect of a fan-in architecture: many publishers and one subscriber. Since each receiver has one active subscriber, they receive messages from only one broker. From the applicaton perspective, there is no difference between Archetype 1 and Archetype 2. They both send and receive messages via their respective virtual cluster with the same result. Archetype 2 is useful in an environment where the reachability of IoT devices is not guaranteed due to poor wireless networks, device mobility and failures, firewalls, and etc. Unlike Archetype 1, which relies on a stable network for the publisher to reach endpoints, Archetype 2 relies on broadcasted messages to reach endpoints regardless of network issues.

clusters:
  - name: edge
    fos: 3
    publisherType: ALL
    subscriberCount: 2
    connection:
      serverURIs: [tcp://localhost:1883-1892]

Archetype 2

Archetype 3

Archetype 3 refers to Archetype 1 that targets endpoints for message delivery. An endpoint can be targeted by its endpoint name or topic base. If topicBase is not specified, then defaultTopicBase is used.

clusters:
  - name: edge
    defaultTopicBase: test/
    endpoints:
      - name: special-node
        endpoint: tcp://localhost:1885
        topicBase: test/special
    connection:
      serverURIs: [tcp://localhost:1883-1892]

Archetype 3

Archetype 4

Archetype 4 creates an incoming bridge. See Bridging Incomming Messages for details.

clusters:
  - name: edge
    connection:
      serverURIs: [tcp://localhost:1883-1892]
    bridges:
      in:
        - cluster: enterprise
          topicFilters: [test/#]
          qos: -1
  - name: enterprise
    connection:
      serverURIs: [tcp://localhost:32001-32005]

Archetype 4

Archetype 5

Archetype 5 creates an outgoing bridge. See Bridging Outgoing Messages for details.

clusters:
  - name: edge
    connection:
      serverURIs: [tcp://localhost:1883-1892]
    bridges:
      out:
        - cluster: enterprise
          topicFilters: [test/#]
          qos: 2
  - name: enterprise
    connection:
      serverURIs: [tcp://localhost:32001-32005]

Archetype 5

Archetype 6

Archetype 6 creates a sticky (proxy) virtual cluster that provides HA at the broker level. See Sticky Clusters for details.

clusters:
  - name: sticky
    fos: 2
    connection:
      serverURIs: [tcp://localhost:31001-31002]

Archetype 6

Archetype 7

Archetype 7 increases the level of availability and scalability of Archetype 6 by including additional broker pairs. Instead of limiting to a single broker pair by Archetype 6, Archetype 7 creates multiple broker pairs to increase the number of brokers.

clusters:
  - name: publisher
    fos: 3
    publisherType: ROUND_ROBIN
    subscriberCount: 0
    connection:
      serverURIs: [tcp://localhost:31001-31010]
  - name: subscriber
    connection:
      serverURIs:
        - tcp://localhost:31001
        - tcp://localhost:31003
        - tcp://localhost:31005
        - tcp://localhost:31007
        - tcp://localhost:31009

Archetype 7

Archetype 8

Archetype 8 combines Archetype 1 (fan-in) and Archetype 2 (fan-out) to provide a butterfly architecture: many publishers and many subscribers. In Archetype 8, the HaMqttClient publishes and subscribes to all the brokers in the virutal cluster. This architecture is useful when edge devices are completely sandboxed and do not have access to MQTT brokers other than their localhost broker.

clusters:
  - name: curator
    publisherType: ALL
    connections:
      - connection:
          serverURIs: [tcp://localhost:1883-1892]

Archetype 8

Archetype 9

Archetype 9 integrates various endpoints in virtual clusters by configuring plugins. There are three (3) types of HaMqttClient plugins: IHaMqttPlugin, IHaMqttConnectorPublisher, and IHaMqttConnectorSubscriber. IHaMqttPlugin is for embedding application logic in virtual clusters. IHaMqttConnectorPublisher and IHaMqttConnectorSubscriber are for creating connectors that can bridge targets other than MQTT brokers. For example, the following diagram shows a virtual cluster writing edge device data to QuestDB and Hazelcast via two subcriber connectors (IHaMqttConnectorSubscriber).

plugins:
  - name: questdb
    description: Stream JSON messages to QuestDB
    enabled: true
    context: CLUSTER
    className: padogrid.mqtt.connectors.QuestDbJsonConnector
    properties:
      - key: endpoint
        value: localhost:9009
    subscriptions:
      - topicFilters: [test/#]
        qos: 1
  - name: hazelcast
    description: Stream JSON messages to Hazelcast
    enabled: true
    context: CLUSTER
    className: padogrid.mqtt.connectors.HazelcastJsonConnector
    properties:
      - key: clusterName
        value: dev
      - key: endpoints
        value: localhost:5701,localhost:5702
      - key: dsType
        value: MAP
      - key: keyType
        value: UUID
    subscriptions:
      - topicFilters: [test/#]
        qos: 1

clusters:
  - name: edge
    pluginNames: [questdb, hazelcast]
    connections:
      - connection:
          serverURIs: [tcp://localhost:1883-1892]

Archetype 9

References

  1. Mosquitto/MQTT Virtual Cluster Tutorial, PadoGrid Bundles, https://github.com/padogrid/bundle-mosquitto-tutorial-virtual-clusters.

◀️ Clustering MQTT :link: Enabling Mosquitto SSL/TLS ▶️