Author: David Mandala Github LinkedIn

In the past there have been at least 8 different types of messages being passed in the mesh network. They were identified by ASCII text strings of various lengths and had to be parsed as a string to figure out what they were, additionally they were not documented anywhere and could be changed by accident when programmers were making changes in the CDP libraries.

In the case of the captive portal message they had several problems. First, they were not versioned in any way that could be programmatically separated. The pages asked about emergency food needs and later it was asked to change the page to ask about medical needs. There was no easy way to tell when each device was cut over, which could result in mixed data that would require a human to manually separate. Also the text data was not properly escaped. The code separated fields with a "\*" but the code never checked the fields to make sure the submission did not contain any "\*" which would have broken the parser at the far end. Also by using ASCII tags it made the message much longer then it needs to be. Since the current LoRa technology messages are limited to 256 characters, space should not be wasted in a message. An example of a status message from the old method is below:

d018avedstatusAara\*Nathee\*726 N Bome
Street\*Portland\*97217\*503xxxxxxx\*ok\*water\*transportation\*food\*inspection\*firstaid\*shelter\*2\*2\*\*1
cat 1 dog\*No additional comments\* \* LEN:182

That is one long ASCII line. At the beginning of the line is a duck identifier, topic type and then the message created from the captive portal on that specific duck. If the captive portal changed, the data would change but it would not be documented anywhere. So if anyone accidentally changed the page, the data could become useless.

As a quick experiment the message was re-coded into a more compressed format: all string info on the captive web portal page was converted to pascal style strings[^1^](#fn1){#fnref1 .footnote-ref}, any radio buttons and check boxes are stored in bit fields. This format massively compressed the string length with no data loss.

d018avedAaraNathee726 N Bome StreetPortland97217?31841 1 cat 1 dogNo
additional comments

Because it is using binary bit fields, the string in a text file looks a bit odd but it can be easily re-expanded programmatically. The new string length is 95 characters long and there is no chance of losing data because someone entered a "\*" as part of their data. The radio buttons were: are you ok or emergency, do you need: water, transportation, food, inspection, first-aid, shelter. Finally a count was requested of how many people are present (adults, children, seniors). This was converted by a technique called bit shifting which allows us to fit 3 numbers into one 16 bit number. The Adult count were shifted 10 bits to the left, the children count were shifted 5 bits to the left and seniors added on the end. By doing this we can account for 31 adults, 31 children and 31 seniors present which is more than enough. Instead of 7+ characters we were able to encode the numbers into 2 characters!

But this is all old history. To save space and to codify message handling and standardize how messages are handled in the future we have a new method since the CDP library code has been changed.

The new method is standardized, compressed, and topics are now single binary characters that can encode up to 256 different topics. Certain topic numbers are reserved for system messages like "are you alive" (PING), and "yes I'm alive" (PONG) and will never be used by any applications. The new message handler will reject any system topics. It is important to keep messages as short as possible. This will help to keep from overloading the mesh and losing messages. Just because 255 topics sounds like a large number it still can be exhausted. It is best practice if the message topics are used frugally, and only added when the topics are really different or you can't combine things via business logic. (Some examples will be provided in this document.)

Sending messages has been greatly simplified for the CDP (if customizations are not needed). Using the CDP you will interact with primarily two fields: the Topic and the message data. Still it is good to have an understanding of messages are formed, most of the information below is informational only and you won't interact with it normally.

Here is the layout of the new message system:

Certain fields are not accessible via business logic and that is intentional. The DUID and the MUID can not be changed via the application level, though the application level can get access to the last messages MUID and there are times that can be very useful and will be explained later in this document.

At the application level you don't need to worry about the low level format of the message. So you simply call the sendData function with the topic and the data that you want sent.

sendData(byte topic, byte data[], byte muid[])

The value of MUID is not important to fill in because the MUID is generated from the initial Duck it originated from. The MUID can be useful in linking sequential messages that were too big to be sent as a single message, that will be explained later in this document.

With the topic field being a byte in length you can store up to 256 "topic" types in the field. This sounds like a lot of topics but they can be used up quite rapidly if you don't think about it. The CDP libraries reserve the first 0x0F values for system messages and are not allowed to be used by application messages. Everything above the reserved numbers are available for application use.

Below is a table of the old topics and what its ID numbers could be. This is not the recommended format:

The problem with the above format is that as you add more and more sensors you use up topic space for no good reason. With a bit of thought, you can use fewer topics up and combine data types.

properly.

By combining all of the Captive Portal Messages and Sensor data into single topics you save the critical topic space for your application whether it be for a business or a personal project. You do have to use business logic to determine what is in your data packet. The CDP libraries define what is at the "system" layer not the business logic application layer. It is the job of the application developer to determine how to send its user case data.

Several examples will be shown below, these are recommended good practices but not mandatory. You can invent your own methods of knowing what is in your data.

Before jumping into the examples below, a quick reminder, the application data chunk of the message is just that "it's under the control of the business application, not the CDP libraries". The designs below are not part of the CDP libraries and do not have to be followed. However, they have been used many times in the past and are well tried methods. Really the key here is that you need easy ways to know what your data packet contains.

If the data packet can only have one kind of data ever, well then you don't need to identify it past the topic level. However that said, the status, captive portal messages, gps and sensor data can be in many different formats and treating them with a "sub topic" ID system to save system level topics makes a lot of sense.

As the first byte of the data, we add a "sub topic" byte that can identify which sensor the data is coming from (this is controlled by the application not the CDP). By having a predefined rigid format for each sub topic type you can always extract the data. In other words, you simply look at the first byte in the message and depending what byte it is, you know how to decode the rest of the message. This format also allows data compression.

By designing the message data like the higher level system message format it is quite possible to extend the 0x13 topic to contain 256 different sensors that can be easily determined in the database the data is sent to. How do we do this?

Something to think about: Do you need to always send the data from the sensor? Say you are sending the data once every 15 minutes, and you are sending temp, humidity and barometric pressure in each packet, do you really need to send the temp if it has not changed since the last 15 minutes? Same with the other data. By careful design you can send a message that can be easily decoded but only sends data that has changed.

The format below is an example for sending Sensor ID type 0x05 (BMP/DHT) data in a message block. (Again this is under application program control not CDP libraries.) It is an example of how data could be put into the message data string. It is really very similar to the format used for the entire message string. The idea is to be able with reading a single byte or two bytes know what is in the data and the format of that data for easy extraction.

Worst case the packet is 6 bytes long if all data is new or all data has changed since the last packet. The data packet would look something like (represented in hex):

0x050x070x230x640x044C = All fields, first sending or all changed

0x050x05x230x044C = Temp and Pressure, Humidity did not change

0x050x00= No fields, no changes. You know the duck is alive

So when the packet arrives the first thing is to check the first byte. The first byte in tells you that you are dealing with BMP/DHT data. Now that you know you what type of data it is, you can format rest of the string. Depending on the bit field you will know how many values that are actually in the data. The order never changes, it's always Temp, Humidity, Pressure, they just pack in if something is not sent.

Now clearly this example is somewhat made up, at most you only save 4 bytes, but just imagine the byte savings if you had 20+ sensors!

The format below is an example for sending Sensor ID type 0x03 (dust) data in a message block. (Again this is under application program control not CDP libraries.) It is an example of how data could be put into the message data string. It is really very similar to the format used for the entire message string. The idea is to be able with reading a single byte or two bytes know what is in the data and the format of that data for easy extraction.

In this case the packet is 3 bytes long, the data is sent every time. The data packet would look something like (represented in hex):

0x030x00ff = All fields sent, dust value 255ug/m3.

0x030x0258 = All fields sent, dust value 600ug/m3.

You could have added a bit field to tell if the data was sent or not but for a 2 byte field adding half again its size to avoid sending the value is not worth it, just send the two bytes and be done.

So when the packet arrives the first thing you do is check the first byte, in this case it tells you that you are dealing with dust data, that tell you for format to decode the rest of the string. In this case just read the next two bytes as a short int and you have the value of the dust sensor.**

*Now clearly this example is somewhat made up, but if you had 20 dust sensors attached all you would do is add more fields and perhaps a bit field to show what was sent and what was not sent, you get the idea. Again how *YOU store YOUR data is your choice, this is just an example of how you can use one system topic to send many different types of sensor data.

As explained above in the sensor data examples you can do much the same thing for captive portal data and be able to have many different captive portal pages being sent in at the same time. Perhaps in one emergency area you are worried about flooding but in a different area you are worried about mud slides. By treating the message data area very much like the example above for sensor data it is possible to have as many captive portal pages (CPP) in use and be able to decode them accurately.

like the message format it is quite possible to use 0x11 topic and then in the data start with a CPM page version to id which captive portal web page is being sent and then have a rigid format that can compress the data. Remember the message data area is under application program control not the system level CDP libraries, this example shows a possible way to save system level topics and be able to have 256 different potential web pages used at the same time.

The format below is for sending Captive Portal Web page type 0x01 data in a message block. It is an example of how data could be put into the data string. The idea is to be able with reading a single byte or two bytes know what is in the data and the format of that data for easy extraction.

The web page has the following fields it in:

The first problem with this web page is that there are no constraints on the text field input and there was no error checking on the web page processing converting the submitted web page into a message. If the first field had been miss used and had 300 characters in it the message would have never gotten through. If each of the variable length character (varchar) fields had 40 characters in the the message never would have been sent. But that is an old problem, in the new system it is quite possible to have as many different web pages as you need, and submit them with their own web page id code. Now they can be processed and tabulated. Here is one way to do it, note all varchar strings can be programmatically truncated if too long. This is a type of a data dictionary, it defines exactly how the message is put together and how to take it back apart for human consumption.

Any web page can be encoded and transmitted, as you build the message data you can truncate fields if necessary, with this design you have have 256 different web pages, need more, increase the 1 byte count of the CP Page ID to 2 bytes and go to town. Radio buttons and check boxes encode nicely into tight storage. Also pull down lists can be encoded in a tiny way. So for example if you know there are only 10 city's that could be in your survey then a single byte can encode that answer. You could change the pets string into check boxes and a count for each checkbox. That could also really reduce the data transmitted. The more you use them on a web page the more data you can transmit tightly.

Example How to cross-reference 2 or more messages together

**There may come a time that you need to send more data then can fit in one message and it's important that the data be associated together. For example, perhaps a community emergency response team (CERT) is canvassing an emergency area doing a quick search for damage after a tornado, they have a captive portal web page (CPWP) to fill in but it is recognized that because of the massive damage in the area they might get on the wrong street and not know it. So there is value in sending both the data they gather and the location where they were standing when they gathered it. Then later back at the database the location data can be examined to make sure the address that was filed in the survey matches where they were actually standing, so it's quite critical that the GPS data be associated with the correct CPWP. How to do this at the application layer?

One method is that the application layer has access to the MUID of a message upon the return of sending the message via the sendData() call. If the application grabs the MUID and then formats a GPS message and embeds the MUID into the data of the GPS message, the application at the database receives a CPWP message from a CERT team, the* definition of this message says that it will be followed by a GPS message that will have the MUID of the CPWP message as it's 2*^nd^ *field. The application then looks for a GPS message around that time in the DB checking the second field for a matching MUID and once found you have the matching messages together. So how could it work.

So first we send a message with the topic set to 0x11 (captive portal web page) with the sub-topic set to 0x04 (CERT quick search damage page). This page once sent we extract the MUID and craft the following:

The format below is an example for sending a topic type 0x12 GPS data with a sub-topic of 0x03 in a message block. (Again this is under application program control not CDP libraries.) It is an example of how data could be put into the message data string. It is really very similar to the format used for the entire message string. The idea is to be able with reading a single byte or two bytes know what is in the data and the format of that data for easy extraction.

As you can see the we sent the topic message of 0x12 with a sub-topic of 0x04 saying it was a CERT quick search damage page, and at the application layer we decided that a GPS page should always be sent with that message and be associated with it. So process the CERT CPWP, send it. After the message was successfully sent we grabbed MUID of the message and put it into a message with the topic of 0x12 (GPS message) with a sub-topic of 0x04 saying to expect first a MUID and then a standard NMEA GGA location string.

Now clearly we could improve on this, there is quite a lot of info in a GGA message that we might not need but for the purpose of this example showing a linked message it works as it is. You could even* *link more messages if you had enough data that had to be associated, you just have to craft the necessary sub-topics and use them.

Footnote:

• Pascal style strings have 1 byte to store the length of the string and then the string. This allows packing fields together with no separators since you know exactly when the next string starts.