This is a high level view of the data flow at different points in the architecture.

See System Architecture for a high level description of this architecture, which won't be repeated here.

Most messages consist of a regular pattern, for example dev/lotus/esp32-1234/sht30/temperature=30.1

There are two variations of this:

dev/lotus/esp32-1234/sht30/temperature/max=100 is the SHT30 reporting that the maximum temperature is 100. This is not a guarantee, its a hint to the UX. At startup, the node informs the MQTT broker of these values.

dev/lotus/esp32-1234/set/sht30/temperature/max = 100 would be a message to set the max parameter to 100. This might be generated by the UX, or by some other module on the same, or a different node.

The documentation should be consistent in how it refers to these fields.

There are also subscription messages which are just a topic

There is currently no unsubscription, but this may be added.

MQTT adds two extra fiekds

Retain - which means the broker should remember the last value sent - this is almost always set to true, QOS: 0=None; 1=At least received once; 2=Sent exactly once

We typically use "1" to make sure messages get through, but note there is some de-duplication in outgoing messages.

These messages are encapsulated and stored in different ways but the basic semantics is always the same.

• /sht30
  • temperature
    • max = 100
• /wifi
  • myhomerouter = secretpassword

A Lora Message might be for example: 11dev/lotus/esp32-1234/sht30/temperature:30.1

Where the first character is the QOS, second is Retain and the rest is topic : value

For subscriptions: the message 11subscribe:dev/lotus/esp32-1234/sht30/temperature would be sent.

The LoRa Mesher packet keeps track of sending and receiving nodes, and gateway maps nodes to subscriptions.

Is just an MQTT message with the topicpath and value, and retain and QOS.

The MQTT message is NOT timestamped, and NOT encoded in JSON.

The logger stores incoming messages in CSV files, one file per day, per output.

They can be retrieved in a browser, or for example by an agent at e.g. https://frugaliot.naturalinnovation/data/dev/lotus/esp8266-fb94bb/sht/temperature/2025-10-15.csv

and looks like

1760486414992,"32.1"
1760486444986,"32.2"

There is no header row, and the first column is the date and time in the standard ISO format.

Retrieving these files requires being logged into the portal, typically in a browser, if there is a need to programmatically access the files we will come up with a way.

There is a configuration file that specifies what topics should be monitored.

The logger can be configured to append any list of topics, on a perioidic basis, to a google sheet such as this which looks like:

The google sheet can contain other content, for example this sheet contains a graph updated live from a node in my office.

It is expected that Graam Disha will be adding Firebase storage as part of their project.

The client (HTML, Javascript, CSS) subscribes to MQTT topics, as shown above, it builds up a hierarchy of Web Components.

<mqtt-project>
  <mqtt-node>      - one per known node
    <mqtt-group>   - equivalent to module on the node
      <mqtt-topic> - equivalent to the input or output on the node

The graphs in the client read CSV files, as shown above, when displaying historical data.

TODO add a diagram

• client changes a value - sends e.g. dev/lotus/esp32-1234/set/xx = yyy
  • TODO check set messages with qos=1 retain=false
• MQTT Broker passes to Logger
• Logger currently stores in CSV, and sends to Forwarders, in future will ignore set messages
• Broker relays this message when it sees the node next
• Node receives and queues it:
  • System_MQTT::messageReceived() -> System_Messages::queueIncoming() -> System_Messages.incoming
• Queue sent to each module
  • System_Messages::dispatchIncomingQueued -> System_Message.dispatch -> System_Frugal::dispatchTwig -> System_Group::dispatchTwig
• Module processes, writes and echoes (as non-set)
  • (module)::dispatchTwig -> System_Base::writeConfigToFSandEcho -> write to FS, and send dev/lotus/esp32-1234/xx=yy retain=true qos=1
• System_Messages::send -> sendRemote & queueLoopback
  • System_Messages::sendRemote queues for outgoing, updating any existing outgoing message
  • System_Messages::queueLoopback puts in incoming queue, handled as above, but note since !isSet will only be picked up by Controls wired to this topic
• System_Messages::sendOutgoingQueued -> System_Message::queuedMessage
• System_Message::queuedMessage, sends via MQTT/WiFi or LoRaMesher to MQTT Broker
• Broker retains it
• Broker does NOT send it back to nodes as they are only subscribed to e.g. dev/lotus/esp32-1234/set/#
• Broker sends it on to any connected UX clients which now has confirmation
• TODO - expand detail of handling on client
• Broker sends to logger
• Logger stores in a file,
• Logger passes to Forwarders
• Forwarders e.g. Google Sheets or Firebase send to those outside apps

A new reading from a sensor

• OUTxxx::set -> IO::send -> System_Messages::send
• System_Messages::send ... see above

Restart of device

• Restart creates modules and calls readConfigFromFS and subscribe e.g. dev/lotus/esp32-1234/set/#
• System_Base::readConfigFromFS -> (module)::dispatchTwig
• (module)::dispatchTwig see above, note flagged as isSet
• note this cause a message to be sent, and retained
• Subscription to set/# may get messages retained from broker (in future these should be !retain qos=1)
• if those messages previously received will be same as stored on disk so not change anything

Connection of client

• subscribes to e.g. dev/lotus
• gets config file and subscribes to nodes in it e.g. dev/lotus/esp32-1234/#
• or sees discovery message e.g. dev/lotus/esp32-1234
• Broker sends retained state which is used to populate UX

Restart of Server & Logger

• Shouldn't effect above except currentValue{} will be cleared and need repopulating as devices restart

Deletion of MQTT Broker memory

• removes all retained messages,
• nodes will have state stored on disk but will not resend to broker until restart
• so broker will not send current state on new UX client connection.

Captive portal

• HTTP POST / with params like frugal_iot/name = "My device"
• -> System_Messages::queueFromCaptive( set/frugal_iot/name = "My device")-> System_Messages::queue_incoming
• see above

Parameters e.g. set/sht/temperature/max=123

• (module)::dispatchTwig -> (all OUT*)::dispatchLeaf
• OUT*::dispatchLeaf -> OUT*::set and writeConfigToFS
• OUT*::set -> IO::send -> System_Messages::send
• see above

LoraMesher - upstream

• System_Message::queuedMessage -> System_Loramesher::publish -> buildAndSend -> Loramesher::send
• queued and sent inside LoraMesher
• System_Loramesher::processReceivedPacket -> System_Messages::send
• see above

LoraMesher - downstream

• System_Message::dispatch -> Frugal::dispatchPath ->
• System_Loramesher::dispatchPath checks if matches any subscription from node, if so -> relayDownstream -> buildAndSend ->
• Loramesher;:send -> queued and sent inside LoraMesher
• System_Loramesher::processReceivedPacket -> System_Messages::queueIncoming
• see above

• wiki
  • Systems Architecture
  • Explanation of configuration files - TO BE WRITTEN