6. Transmitting data over MQTT - ControlBits/EMIT GitHub Wiki
Now we have a fully functional Environmental Sensor that is connected to the internet, we're going to tackle how to transmit data across the internet using the MQTT protocol.
MQTT is a light-weight publish/subscribe messaging protocol designed for the transmission of data where there is limited bandwidth or unreliable connectivity. This makes it ideal for small sensors like EMIT.
When we're finished we'll be able to see our sensor data, in real-time, on the EMIT 'LIVE' dashboard.
EMIT 'LIVE' is a free, real-time Environmental Monitoring dashboard we've created specifically for the EMIT project.
We’ve already done the coding and integration to MQTTHQ.com (the projects free public MQTT broker) so that anyone using or developing with EMIT can immediately test and use their hardware without having to write any test or application code.
OK, let's get started...
6.1 Encoding our data using 'JSON'
In order to make our data easier to read, transmit and decode at the other end, we need to encode our data using 'JavaScript Object Notation' or 'JSON'. Wikipedia.com defines JSON as:
JavaScript Object Notation (JSON, pronounced /ˈdʒeɪsən/; also /ˈdʒeɪˌsɒn/) is an open standard file format, and data interchange format, that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and array data types (or any other serializable value). It is a very common data format, with a diverse range of applications, such as serving as a replacement for XML in AJAX systems.
Now this might sound complicated but in reality, JSON is just a string that contains the timestamp and environmental data we are already creating ... it's just re-formatted using some pre-defined rules.
In our case, we're going to encode our data in 'attribute-value' pairs as follows:
{
"TimeUTC" : "2020-08-05 13:53:39",
"TemperatureC" : "22.6",
"Humidity" : "66.3"
}
Where the 'attributes' are our variable names (e.g. 'TimeUTC') and the 'values' are our Timestamp or measurements expressed as a string.
While the above JSON example is easily readable, we don't need or require the white-space that helps 'us humans' read it ... that just takes up bandwidth in our communications. So when we transmit our data, we'll strip out all of the white-space so that it looks like this:
{"TimeUTC":"2020-08-05 13:53:39","TemperatureC":"22.6","Humidity":"66.3"}
Our 'EMIT LIVE' dashboard, has been designed to read and process JSON data received in this format, so it is important that the JSON we send it via MQTT (the subject of the next section) is exactly as detailed above including the attribute names.
Creating our JSON is just a simple string concatenation, the only complication is that we need to maintain the double quotation (" ") makes within the JSONstring.
As a result we use single quotation marks to define our string parts and use the plus sign to concatenate in our variables as follows:
JSONstring = '{"TimeUTC":"'+get_time_str()+'","TemperatureC":"'+str(tempC)+'","Humidity":"'+str(humidity)+'"}'
Notice how we are using 'get_time_str()' to give us the timestamp, and using the str() function to convert the 'tempC' and 'humidity' variables to strings (from floating point numbers).
To check the concatenation has worked properly, we can print it to the Shell using:
print(JSONstring)
When transmitting data between systems it good practice to transmit time in 'UTC' (Coordinated Universal Time) and all other measurements in their 'SI' (International System of Units) units.
In our case, our NTC synchronised timestamp is already in UTC and we'll transmit the temperature in Celcius and Humidity in '%RH' (% Relative Humidity).
Following this convention we can also clean up our code by deleting the following redundant code from main.py
print("Reading AM2302 ...")
tempF = (tempC * 9/5) + 32.0 # convert Celcius result into Fahrenheit
print("Time Stamp: " + get_time_str())
print("Temperature (C): " + str(tempC))
print("Temperature (F): " + str(tempF))
print("Humidity (%RH): " + str(humidity))
... and replacing it with the new JSON code we created above.
Our new main.py now looks like this:
# connect to wifi
wifi_connect()
# wait 5 seconds for network to settle
time.sleep(5)
# synchronise local clock with NTP time
sync_to_NTP()
# this is the main program loop
while True:
time.sleep(5) # wait 5 seconds
RedLED.value(1) # turn RedLED ON
AM2302.measure() # start AM2302 measurement
RedLED.value(0) # turn RedLED OFF
tempC = AM2302.temperature() # get temperature (Celsius) from AM2302
humidity = AM2302.humidity() # get humidity from AM2302
JSONstring = '{"TimeUTC":"'+get_time_str()+'","TemperatureC":"'+str(tempC)+'","Humidity":"'+str(humidity)+'"}'
print(JSONstring)
6.2 Sending data using MQTT
In order to send data using MQTT, we are going to use the 'umqtt.simple2' client for MicroPython. This client is significantly more robust than the standard MicroPython MQTT client as it has better error handling which allows us to create a robust connect/re-connect process to handle the normal day-to-day communication issues such as poor Wi-Fi signal strength, temporary network drop-outs, etc.
As the 'umqtt.simple2' isn't part of the standard MicroPython distribution, you'll need to download it to your PC first and then upload it to your EMIT's ESP32.
6.2.1 Download 'umqtt.simple2'
There are several ways to download the code, described at the projects GitHub page: umqtt.simple2 GitHub repository. In our case, we're going to download the code manually and then upload it to EMIT using Thonny. Here goes...
1. Download the code to your PC - Simply click the 'Download Code' button on the projects GitHub page and choose 'Download ZIP'
2. Un-ZIP the files on your PC
3. Create a new directory on EMIT's ESP32 as follows:
Step 1: In Thonny, click the 'File Open' icon and select 'MicroPython Device'
Step 2: In the 'Open from MicroPython device' dialog, right-click in the file window and select 'New directory...'
Step 3: A 'New directory' dialog will appear. Enter the name of the new directory - in our case 'umqtt'
If successful, the new directory should appear in the 'Open from MicroPython device' dialog. Click cancel to close the dialog.
4. Open the 'simple2.py' file in Thonny - we're going to use the 'src_minimized' version to save on memory.
5. Save the 'simple2.py' file to the new directory you just created on EMIT's ESP32 - from Thonny's main menu, select 'Save as', choose 'MicroPython device' and in the 'Open MicroPython device' dialog, double click the umqtt directory and enter the name of the file 'simple2.py'.
6.2.2 Set up the MQTT client in boot.py
OK, now we have everything we need, let's set up our MQTT client in boot.py.
First we need to import the MQTT client 'MQTTClient' from the umqtt.simple2 module that we've just uploaded to EMIT's ESP32 (Note: the 'umqtt.' notation represents the directory we've just put 'simple2.py' into).
We'll also need to import the 'ubinascii' module which will be used later to convert strings into byte arrays, ready for transmission via MQTT.
from umqtt.simple2 import MQTTClient
import ubinascii
It is estimated, that there are already more than 25 Billion IoT devices connected to the internet! When using MQTT, every new device needs to have a unique identifier.
To create a unique ID for our EMIT device we're going to create a UUID (Universally Unique IDentifier) using a free web application at: https://www.uuidgenerator.net. To get your own UUID simply click on the link to visit the UUID Generator website and copy the UUID it automatically generates for you to your clip-board by clicking the 'Copy' button.
Then all you need to do is past it into your code and use it to define a variable called 'deviceUUID '. This needs to be a text string so just place a single quotation at the start and finish. If should look something like this (DO NOT use this one as it's already in use by the EMIT development board used to create this guide!):
deviceUUID = '1116fa09-973c-4fa3-aff1-3f9859c819ff'
Now we have a UUID for our EMIT, we're going to use this to as our 'Client ID' (variable name: client_id) and also as our MQTT 'topic'.
Our 'client_id' needs to be a hexadecimal representation of our deviceUUID string, we can generate this easily using the ubinascii.hexlify() function we imported above.
Our 'topic' needs to be a byte array representation of our deviceUUID string which we can generate using the standard bytes() conversion function built into MicroPython.
client_id = ubinascii.hexlify(deviceUUID)
topic = bytes(deviceUUID, 'utf-8')
The last settings we need to define is the name of our MQTT broker and the port through which our broker will accept our messages.
For this guide we're going to use the free MQTTHQ.com broker created specifically for the EMIT project. MQTTHQ is a load-balanced, multi-node MQTT broker cluster, designed to provide a stable, highly-reliable broker for development and production purposes.
MQTTHQ supports both TCP and WebSocket connections, in this case we'll be using TCP.
We define the MQTT client settings (taken from the MQTTHQ.com website) as follows:
mqtt_broker = 'public.mqtthq.com'
mqtt_port = 1883
OK, now we have our settings, we can define our MQTT broker connection function 'mqtt_connect()'.
We want this function to attempt to connect to the broker using the settings we've just defined. If it can't connect to the broker after 5 attempts (which is probably due to a loss of Wi-Fi connection) we'll reboot the ESP32 - forcing a full re-connect process including Wi-Fi.
The finished function is shown below. As everything is fairly straight forward, rather than walk through it here, we've added lots of comments within the code itself.
The only MQTT specific lines to note are the definition of the MQTTClient object which is done like this ...
client = MQTTClient(client_id, mqtt_broker, mqtt_port)
... and the command need to ask the client to connect to the broker:
client.connect()
The full code for the 'mqtt_connect()' function is:
# define function to connect to MQTT broker
def mqtt_connect():
global client_id, mqtt_broker, mqtt_port # use the global variables: client_id, mqtt_broker, mqtt_port
client = MQTTClient(client_id, mqtt_broker, mqtt_port) # create MQTT client using our MQTT settings
# initialise our local variables (used to manage the connection process)
broker_connected = 0 # broker connection status register
connect_attempt = 1 # keep track of the number of connection attempts
max_con_attempts = 5 # set the maximum number of connection attempts - we'll reboot after this number of attempts
while (connect_attempt <= max_con_attempts) and broker_connected == 0:
print('Connecting to MQTT broker: '+mqtt_broker+', publishing to topic: ' + topic.decode("utf-8") + 'Attempt No. :' + str(connect_attempt))
try:
client.connect() # attempt to connect to the MQTT broker
print("Connection successful") # if successful, send success message to Shell
broker_connected = 1 # set broker connection status register to 1
except:
print("Connection attempt failed") # if NOT successful, send debug message to Shell
if connect_attempt < max_con_attempts: # if current connection attempt is less than max connection attempts ...
connect_attempt = connect_attempt + 1 # increment the connection_attempt count
time.sleep(2) # wait 2 seconds before re-attempting a connection
else:
print('Failed to connect to MQTT broker after ' + str(connect_attempt) + ' attempts. Rebooting...') # if max_con_attempts reached ...
time.sleep(2) # wait 2 seconds
machine.reset() # reset the ESP32 ... forcing a full re-connection process including Wi-Fi
return client # if connection successful, return the client object
With the new function added, our complete boot.py should look like:
# general board setup
import machine, time, dht
import network, ntptime
from umqtt.simple2 import MQTTClient
import ubinascii
# configure GPIO pins
RedLED = machine.Pin(16, machine.Pin.OUT)
RedLED.value(0)
GreenLED = machine.Pin(17, machine.Pin.OUT)
GreenLED.value(0)
AM2302 = dht.DHT22(machine.Pin(14))
# define Wi-Fi settings
wifiSSID = 'your-WiFi-SSID-goes-here'
wifiPasswd = 'your-WiFi-password-goes-here'
# Please generate a uniquie UUID at: https://www.uuidgenerator.net/ and replace the one below
# Don't forget the single quotations start and finish!
deviceUUID = '1116fa09-973c-4fa3-aff1-3f9859c819ff'
# define MQTT settings
client_id = ubinascii.hexlify(deviceUUID)
topic = bytes(deviceUUID, 'utf-8')
mqtt_broker = 'public.mqtthq.com'
mqtt_port = 1883
# define function for setting up Wi-Fi network
def wifi_connect():
wifi = network.WLAN(network.STA_IF) # create our 'wifi' network object
wifi.active(True) # turn on the Wi-Fi hardware
# if not connected ...
while wifi.isconnected() == False:
GreenLED.value(1) # turn Green LED ON
print('trying WiFi connection ', wifiSSID) # print 'trying..' message to Shell
wifi.connect(wifiSSID, wifiPasswd) # try connecting to wifi
time.sleep(1) # wait 1 second
GreenLED.value(0) # turn Green LED OFF
time.sleep(2) # wait 2 second
# if connected ...
GreenLED.value(1) # turn Green LED ON
print('WiFi connection successful') # print 'WiFi connection successful' message to Shell
print(wifi.ifconfig()) # print WiFi network settings (inc. IP Address) to Shell
# define function to synchronise local clock with NTP time
def sync_to_NTP():
ntpConnected = 0 # initialise an 'ntpConnected' variable to hold status, set it to
while ntpConnected == 0: # while not connected to NTP network ...
try:
print('Trying to Sync with NTP servers') # print debug message to Shell
ntptime.settime() # set localtime to NTP time
ntpConnected = 1 # set ntpConnected status to 1
except:
print ('NTP error') # print debug message to Shell
ntpConnected = 0 # set ntpConnected status to 0
print('Local time synchronised: ', get_time_str()) # print the new localtime as a string
# define function for getting local time as string
def get_time_str():
my_time = time.localtime() # get 'localtime' and store it in a variable 'my_time'
my_time_string = '{:04d}-{:02d}-{:02d} {:02d}:{:02d}:{:02d}'.format(my_time[0], my_time[1], my_time[2], my_time[3], my_time[4], my_time[5]) # convert localtime to a formatted text string 'my_time_string'
return my_time_string # return the formatted string
# define function to connect to MQTT broker
def mqtt_connect():
global client_id, mqtt_broker, mqtt_port # use the global variables: client_id, mqtt_broker, mqtt_port
client = MQTTClient(client_id, mqtt_broker, mqtt_port) # create MQTT client using our MQTT settings
# initialise our local variables (used to manage the connection process)
broker_connected = 0 # broker connection status register
connect_attempt = 1 # keep track of the number of connection attempts
max_con_attempts = 5 # set the maximum number of connection attempts - we'll reboot after this number of attempts
while (connect_attempt <= max_con_attempts) and broker_connected == 0:
print('Connecting to MQTT broker: '+mqtt_broker+', publishing to topic: ' + topic.decode("utf-8") + 'Attempt No. :' + str(connect_attempt))
try:
client.connect() # attempt to connect to the MQTT broker
print("Connection successful") # if successful, send success message to Shell
broker_connected = 1 # set broker connection status register to 1
except:
print("Connection attempt failed") # if NOT successful, send debug message to Shell
if connect_attempt < max_con_attempts: # if current connection attempt is less than max connection attempts ...
connect_attempt = connect_attempt + 1 # increment the connection_attempt count
time.sleep(2) # wait 2 seconds before re-attempting a connection
else:
print('Failed to connect to MQTT broker after ' + str(connect_attempt) + ' attempts. Rebooting...') # if max_con_attempts reached ...
time.sleep(2) # wait 2 seconds
machine.reset() # reset the ESP32 ... forcing a full re-connection process including Wi-Fi
return client # if connection successful, return the client object
6.2.2 Transmitting data using our MQTT client in main.py
We've now completed the hard work ... setting up the MQTT client. Using the client to transmit data in our main.py application code is now easy!
First, we need to call the mqtt_connect() function to create our MQTT broker connection.
This is done (after a 1-second delay) immediately after we've synchronised our localTime to the NTP servers. This is done by inserting these 2 lines into main.py after the 'sync_to_NTP()' call:
time.sleep(1)
client = mqtt_connect()
We're now ready (finally!) to send some data. We've already created the 'JSONstring' that we're going to send, but before we can transmit it over MQTT we need to convert it to a byte array. This is done in the same way as we converted our 'topic' above - using the standard bytes() conversion function built into MicroPython.
MQTTmsg = bytes(JSONstring, 'utf-8')
Then all that's left is to publish our 'MQTTmsg' to the 'topic' we're going to create via the broker. The topic is automatically created the first time it is published to. We publish our MQTT message using the 'client.publish()' function as follows:
client.publish(topic, MQTTmsg)
BUT there are lots of reasons why the publish might fail (many of them beyond our control) such as a temporary drop out of the broadband connection into the building, a short-term restriction in network bandwidth, etc.
As a result, in order to build a resilient IoT device, we need to build in some robust error handling. We'll do this using the 'try - except' method; where if we can't publish our MQTTmsg, we'll attempt to re-connect to the broker using our 'mqtt_connect()' function - which already has the reboot/reconnect process built in to it. All this is done as follows:
try:
client.publish(topic, MQTTmsg)
except:
print("Can not publish message - reconnecting")
mqtt_connect()
With the new MQTT publishing code added, our complete main.py should look like this:
# connect to wifi
wifi_connect()
# wait 5 seconds for network to settle
time.sleep(5)
# synchronise local clock with NTP time
sync_to_NTP()
# wait 1 second
time.sleep(1)
# set up MQTT connection and subscribe
client = mqtt_connect()
# this is the main program loop
while True:
time.sleep(5) # wait 5 seconds
RedLED.value(1) # turn RedLED ON
AM2302.measure() # start AM2302 measurement
RedLED.value(0) # turn RedLED OFF
tempC = AM2302.temperature() # get temperature (Celsius) from AM2302
humidity = AM2302.humidity() # get humidity from AM2302
JSONstring = '{"TimeUTC":"'+get_time_str()+'","TemperatureC":"'+str(tempC)+'","Humidity":"'+str(humidity)+'"}'
print(JSONstring)
MQTTmsg = bytes(JSONstring, 'utf-8')
try:
client.publish(topic, MQTTmsg)
except:
print("Can not publish message - reconnecting")
mqtt_connect()
6.2.2 Testing everything with the EMIT 'LIVE' dashboard.
OK, now it's time to upload our boot.py and main.py and do some testing!
To help with this, in order to avoid you having to create some application code to connect to the broker, subscribe to your new topic, display the data, etc ... you never are quite sure if any bugs are in your firmware or your subscriber application! ... we've create EMIT 'LIVE' a real-time Environmental Monitoring dashboard for the EMIT project.
We’ve already done the coding and integration to MQTTHQ.com so all you need to do (once your EMIT firmware is uploaded and running) is head over to https://controlbits.com/emit/live/ and paste your deviceUUID into the 'EMIT UUID/Topic name:' box and click the 'View my EMITs data' button.
All being well, you should now see a dashboard showing your MQTT data ... updated every 5-seconds or so!
Notes:
- If nothing appears on the graphs, check the 'Connection Status' box (bottom-right), you may need to refresh your browser a few times to get your browser to connect to the MQTTHQ broker.
- The relay data will be saying 'undefined' ... simply because we've not implemented our relay code yet ... that's our next task: 7. Using the relay for control