6. Transmitting data over MQTT (Arduino) - ControlBits/EMIT GitHub Wiki
Now we have a fully functional Environmental Sensor that is connected to the internet, we're going to tackle how to transmit data across the internet using the MQTT protocol.
MQTT is a light-weight publish/subscribe messaging protocol designed for the transmission of data where there is limited bandwidth or unreliable connectivity. This makes it ideal for small sensors like EMIT.
When we're finished we'll be able to see our sensor data, in real-time, on the EMIT 'LIVE' dashboard.
EMIT 'LIVE' is a free, real-time Environmental Monitoring dashboard we've created specifically for the EMIT project.
We’ve already done the coding and integration to MQTTHQ.com (the projects free public MQTT broker) so that anyone using or developing with EMIT can immediately test and use their hardware without having to write any test or application code.
OK, let's get started...
In order to make our data easier to read, transmit and decode at the other end, we need to encode our data using 'JavaScript Object Notation' or 'JSON'. Wikipedia.com defines JSON as:
JavaScript Object Notation (JSON, pronounced /ˈdʒeɪsən/; also /ˈdʒeɪˌsɒn/) is an open standard file format, and data interchange format, that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and array data types (or any other serializable value). It is a very common data format, with a diverse range of applications, such as serving as a replacement for XML in AJAX systems.
Now this might sound complicated but in reality, JSON is just a string that contains the timestamp and environmental data we are already creating ... it's just re-formatted using some pre-defined rules.
In our case, we're going to encode our data in 'attribute-value' pairs as follows:
{
"TimeUTC" : "2020-08-05 13:53:39",
"TemperatureC" : "22.6",
"Humidity" : "66.3"
}Where the 'attributes' are our variable names (e.g. 'TimeUTC') and the 'values' are our Timestamp or measurements expressed as a string.
While the above JSON example is easily readable, we don't need or require the white-space that helps 'us humans' read it ... that just takes up bandwidth in our communications. So when we transmit our data, we'll strip out all of the white-space so that it looks like this:
{"TimeUTC":"2020-08-05 13:53:39","TemperatureC":"22.6","Humidity":"66.3"}Our 'EMIT LIVE' dashboard, has been designed to read and process JSON data received in this format, so it is important that the JSON we send it via MQTT (the subject of the next section) is exactly as detailed above including the attribute names.
Creating our JSON is just a simple string concatenation, the only complication is that we need to maintain the double quotation (" ") makes within the JSONstring, so we will put a \ preceding the " to escape it as follows:
snprintf(JSONstring,150,
"{\"TimeUTC\":\"%s\",\"TemperatureC\":\"%f\",\"Humidity\":\"%f\"}",
get_time_str(),tempC,humidity);Notice how we are using 'get_time_str()' to give us the timestamp, and JSONstring is a globaly defined array of 150 character.
// array holding the json string holding the timestamped readings
char JSONstring[150];To check the concatenation has worked properly, we can print it to the Shell using:
Serial.println(JSONstring);When transmitting data between systems it good practice to transmit time in 'UTC' (Coordinated Universal Time) and all other measurements in their 'SI' (International System of Units) units.
In our case, our NTC synchronised timestamp is already in UTC and we'll transmit the temperature in Celcius and Humidity in '%RH' (% Relative Humidity).
Following this convention we can also clean up our code by deleting the following redundant code from main.py
// Print the read values on the Serial Monitor.
Serial.print("Time Stamp: ");
Serial.println(get_time_str());
Serial.print("Temperature (C): ");
Serial.println(tempC);
Serial.print("Temperature (F): ");
Serial.println(humidity);
Serial.print("Humidity (%RH): ");
Serial.println(humidity);... and replacing it with the new JSON code we created above.
Our new Sketch now looks like this:
// WiFi libary is essential for connecting to WIFI
#include <WiFi.h>
// time library sync EMIT ESP32 clock with a NTP server
#include "time.h"
// include the Adafruit DHT libraries to use its functions reading the temperature and humidity from the DHT22 sensor.
#include "DHT.h"
#define EMIT_RED_LED 16
#define EMIT_GREEN_LED 17
#define EMIT_DHT_PIN 14 // Digital pin connected to the DHT sensor
#define DHTTYPE DHT22 // DHT 22 (AM2302), AM2321
// Initialize DHT sensor.
DHT dht(EMIT_DHT_PIN, DHTTYPE);
// define Wi-Fi settings
#define WIFISSID "your-WiFi-SSID-goes-here"
#define WIFIPASSWORD "your-WiFi-password-goes-here"
// we will request the time from pool.ntp.org
const char* ntpServer = "pool.ntp.org";
// The gmtOffset_sec variable defines the offset in seconds between your time zone and GMT
const long gmtOffset_sec = 0;
// defines the offset in seconds for daylight saving time. It is generally one hour, that corresponds to 3600 seconds
const int daylightOffset_sec = 3600;
// array holding the json string holding the timestamped readings
char JSONstring[150];
void wifi_connect(void){
// change the wifi interface mode to station
WiFi.mode(WIFI_STA);
// attempt a WiFi connection to the router with credentials of WIFISSID and WIFIPASSWORD
WiFi.begin(WIFISSID, WIFIPASSWORD);
// wait for the connection to be established.
while (WiFi.status() != WL_CONNECTED){
// flash the Green LED indication a wifi connection attempt.
// turn on the Green LED
digitalWrite(EMIT_GREEN_LED , HIGH);
// print 'trying..' message to Serial Monitor
Serial.print("trying WiFi connection ");
Serial.println(WIFISSID);
// wait 1 second (1000 milliseconds)
delay(1000);
// turn off the Green LED
digitalWrite(EMIT_GREEN_LED , LOW);
// wait 2 second (2000 milliseconds)
delay(2000);
}
// if connected ...
// turn on the Green LED
digitalWrite(EMIT_GREEN_LED , HIGH);
// print 'WiFi connection successful' message to Serial Monitor
Serial.println("WiFi connection successful") ;
// print WiFi network settings (inc. IP Address) to Serial Monitor.
Serial.println("IP address: ");
Serial.println(WiFi.localIP());
}
// define function for getting local time as string
// returns a pointer to a string containing the formatted time.
char* get_time_str(void){
// string holding well formatted time.
static char my_time_string[60];
// structure holding the time
// definition is as follow
// struct tm
// {
// int tm_sec; // seconds from 0 to 59
// int tm_min; // minutes from 0 to 59
// int tm_hour; // hours from 0 to 23
// int tm_mday; // day of the month from 1 to 30
// int tm_mon; // month of the year from 0 to 11
// int tm_year; // number of years from 1900
// ... other variables not in the scope of the tutorial..
// };
struct tm my_time;
// read the time into timeinfo structure.
if(!getLocalTime(&my_time)){
Serial.println("Failed to obtain time");
return NULL;
}
snprintf(my_time_string, // string to add the formated time to.
60, // string length to prevent overflow
"%04d-%02d-%02d %02d:%02d:%02d", // time string format
(my_time.tm_year + 1900),
(my_time.tm_mon + 1),
my_time.tm_mday,
my_time.tm_hour,my_time.tm_min,my_time.tm_sec);
// return the formatted time string
return my_time_string;
}
void setup() {
// Enable the serial terminal
// Useful for debugging messages
Serial.begin(115200);
// configure RED LED GPIO to be OUTPUT
// so that we would control the LED (ON , OFF)
pinMode(EMIT_RED_LED , OUTPUT);
// on the beginning we will turn OFF the RED LED
digitalWrite(EMIT_RED_LED , LOW);
// configure Green LED GPIO to be OUTPUT
// so that we would control the LED (ON , OFF)
pinMode(EMIT_GREEN_LED , OUTPUT);
// on the beginning we will turn OFF the GREEN LED
digitalWrite(EMIT_GREEN_LED , LOW);
// initialize the DHT communication
dht.begin();
// attempt a wifi connection
wifi_connect();
// wait 5 seconds ( 5000 milliseconds ) for network to settle
delay(5000);
// synchronise time with the NTP server.
configTime(gmtOffset_sec, daylightOffset_sec, ntpServer);
}
void loop() {
//Wait for 5 seconds ( 5000 milliseconds )
delay(5000);
// turn the red LED ON
digitalWrite(EMIT_RED_LED , HIGH);
// Print a debugging message on the Serial Terminal indicating the start of the read attempt.
Serial.println("Reading AM2302 ...");
// Reading temperature or humidity takes about 250 milliseconds!
// Sensor readings may also be up to 2 seconds
float humidity = dht.readHumidity();
// Read temperature as Celsius (the default)
float tempC = dht.readTemperature();
// Read temperature as Fahrenheit (isFahrenheit = true)
float tempF = dht.readTemperature(true);
// turn the red LED off indicating the end of the read cycle
digitalWrite(EMIT_RED_LED , LOW);
snprintf(JSONstring,150,
"{\"TimeUTC\":\"%s\",\"TemperatureC\":\"%f\",\"Humidity\":\"%f\"}",
get_time_str(),tempC,humidity);
// Print the read values on the Serial Monitor.
Serial.println(JSONstring);
}In order to send data using MQTT, we are going to use the 'pubsubclient' client library by knolleary.
We will use the library manager to download the pubsubclient library same as stated in 3.1 Libraries Installation. In a brief we will search for the pubsubclient library in the search bar of the library manager.
OK, now we have everything we need, let's set up our MQTT client in .
First we need to include the MQTT client 'Pubsubclient' that we've just installed to Arduino's libraries using the libraries manager.
#include <PubSubClient.h>We will also need to include the WiFiClient library that the MQTT client would use its object to establish a connection with the broker:
#include <WiFiClient.h>It is estimated, that there are already more than 25 Billion IoT devices connected to the internet! When using MQTT, every new device needs to have a unique identifier.
To create a unique ID for our EMIT device we're going to create a UUID (Universally Unique IDentifier) using a free web application at: https://www.uuidgenerator.net. To get your own UUID simply click on the link to visit the UUID Generator website and copy the UUID it automatically generates for you to your clip-board by clicking the 'Copy' button.
Then all you need to do is past it into your code and use it to define a constant character array called 'deviceUUID '. This needs to be a text string so just place a single quotation at the start and finish. If should look something like this (DO NOT use this one as it's already in use by the EMIT development board used to create this guide!):
const char* deviceUUID = "f1f2c71d-3164-4edc-a0a4-75aa172912d0";Now we have a UUID for our EMIT, we're going to use this to as our 'Client ID' and also as our MQTT 'topic'.
The last settings we need to define is the name of our MQTT broker and the port through which our broker will accept our messages.
For this guide we're going to use the free MQTTHQ.com broker created specifically for the EMIT project. MQTTHQ is a load-balanced, multi-node MQTT broker cluster, designed to provide a stable, highly-reliable broker for development and production purposes.
MQTTHQ supports both TCP and WebSocket connections, in this case we'll be using TCP.
We define the MQTT client settings (taken from the MQTTHQ.com website) as follows:
const char* mqtt_broker = "public.mqtthq.com";
int mqtt_port = 1883;Then we will define the MQTT client object that uses a WiFiClient object to establish the MQTT connection to the broker.
WiFiClient emitClient;
PubSubClient mqttclient(emitClient);Then we need to initialize the MQTT client in setup() with its parameters which are the broker name, port and events callback function. the callback function is meant to be called by the client on reception of a new message which we would need on a later stage of that tutorial.
// initialize the MQTT client with the broker name and port number
mqttclient.setServer(mqtt_broker, mqtt_port);
// set the reception callback of the MQTT client to be called by the client stack on reception of a new message.
mqttclient.setCallback(mqttclient_callback);OK, now we have our settings, we can define our MQTT broker connection function mqttclient_connection(). the function will be called in loop as it take care of the whole MQTT client life cycle.
We want this function to establish a non blocking attempt to MQTT connection to the broker using the settings we've just defined. if it fails to connect it will return an error status(false) that would trigger another connection attempt after 5 seconds using a simple Arduino timer.
The finished function is shown below. As everything is fairly straight forward, rather than walk through it here, we've added lots of comments within the code itself.
The only MQTT specific lines to note are mqttclient.connected() which returns a boolean representing the connection status of the MQTT client (true or false). and the mqttclient.connect(deviceUUID) function that uses the deviceUUID as a unique client ID for the connection.
The full code for the 'mqttclient_connection()' function is:
// * handles the mqtt client connection.
// * Attempt to connect to the mqtt broker and if it fails.
// * or if its not connected or failed to connect it initiate a timer to reconnect on 5 seconds.
// * call mqttclient.loop which needed to be called continuously in loop for the mqtt client to
// function propaply as it handles the keep alive packets the the broker and check for a received message, etc..
void mqttclient_connection(void) {
// check if the MQTT client is connected or not
if (!mqttclient.connected()) {
// not connected then start a timer to try to reconnect after 5 seconds
long now = millis(); // now holds the current time.
// check if 5 seconds is ellapsed.
if (now - lastReconnectAttempt > 5000) {
lastReconnectAttempt = now; // update the last connection attempt time with the current time to try again after 5 seconds if failed
// Attempt to reconnect
// try to connect to the broker..
mqttclient.connect(deviceUUID);
if (mqttclient.connected()) {
// only if connected successfully stop the timer
lastReconnectAttempt = 0;
}
}
} else {
// Client connected then call the mqttclient.loop that maintain the keep alive messages with the broker, check for a received message and etc..
mqttclient.loop();
}
}Notice that we have implemented a simple timer mechanism that's very famous and handy in Arduino. the timer depends on millis() function which hold the number of milliseconds since the last reset. the timer mechanism checks if current millis minus the last connection attempt millis is more than 5 seconds (5000 milliseconds) and if so it will reset the timer and attempt a new connection. if the connection is successful the timer is stoped by reseting the variable holding the last connection attempt.
to implement the timer a global variable is needed that's holding the last connection attempt millis. Its type is long as it would hold big numbers.
long lastReconnectAttempt = 0;The timer implementation is as follows:
long now = millis(); // now holds the current time.
// check if 5 seconds is ellapsed.
if (now - lastReconnectAttempt > 5000) {
lastReconnectAttempt = now; // update the last connection
// Do a scheduled job..
}With the new function added, our complete Sketch should look like:
// WiFi library is essential for connecting to WIFI
#include <WiFi.h>
// contains the TCP client object need to establish a MQTT connection with the MQTT broker.
#include <WiFiClient.h>
// MQTT client C++ implementation library
#include <PubSubClient.h>
// time library sync EMIT ESP32 clock with a NTP server
#include "time.h"
// include the Adafruit DHT libraries to use its functions reading the temperature and humidity from the DHT22 sensor.
#include "DHT.h"
#define EMIT_RED_LED 16
#define EMIT_GREEN_LED 17
#define EMIT_DHT_PIN 14 // Digital pin connected to the DHT sensor
#define DHTTYPE DHT22 // DHT 22 (AM2302), AM2321
// Initialize DHT sensor.
DHT dht(EMIT_DHT_PIN, DHTTYPE);
// define Wi-Fi settings
#define WIFISSID "your-WiFi-SSID-goes-here"
#define WIFIPASSWORD "your-WiFi-password-goes-here"
// we will request the time from pool.ntp.org
const char* ntpServer = "pool.ntp.org";
// The gmtOffset_sec variable defines the offset in seconds between your time zone and GMT
const long gmtOffset_sec = 0;
// defines the offset in seconds for daylight saving time. It is generally one hour, that corresponds to 3600 seconds
const int daylightOffset_sec = 3600;
// array holding the json string holding the timestamped readings
char JSONstring[150];
// MQTT
const char* deviceUUID = "f1f2c71d-3164-4edc-a0a4-75aa172912d0";
const char* mqtt_broker = "public.mqtthq.com";
int mqtt_port = 1883;
WiFiClient emitClient;
PubSubClient mqttclient(emitClient);
long lastReconnectAttempt = 0;
void mqttclient_callback(char* topic, byte* payload, unsigned int length) {
// handle message arrived
}
// * handles the mqtt client connection.
// * Attempt to connect to the mqtt broker and if it fails.
// * or if its not connected or failed to connect it initiate a timer to reconnect on 5 seconds.
// * call mqttclient.loop which needed to be called continuously in loop for the mqtt client to
// function propaply as it handles the keep alive packets the the broker and check for a received message, etc..
void mqttclient_connection(void) {
// check if the MQTT client is connected or not
if (!mqttclient.connected()) {
// not connected then start a timer to try to reconnect after 5 seconds
long now = millis(); // now holds the current time.
// check if 5 seconds is ellapsed.
if (now - lastReconnectAttempt > 5000) {
lastReconnectAttempt = now; // update the last connection attempt time with the current time to try again after 5 seconds if failed
// Attempt to reconnect
// try to connect to the broker..
mqttclient.connect(deviceUUID);
if (mqttclient.connected()) {
// only if connected successfully stop the timer
lastReconnectAttempt = 0;
}
}
} else {
// Client connected then call the mqttclient.loop that maintain the keep alive messages with the broker, check for a received message and etc..
mqttclient.loop();
}
}
void wifi_connect(void){
// change the wifi interface mode to station
WiFi.mode(WIFI_STA);
// attempt a WiFi connection to the router with credentials of WIFISSID and WIFIPASSWORD
WiFi.begin(WIFISSID, WIFIPASSWORD);
// wait for the connection to be established.
while (WiFi.status() != WL_CONNECTED){
// flash the Green LED indication a wifi connection attempt.
// turn on the Green LED
digitalWrite(EMIT_GREEN_LED , HIGH);
// print 'trying..' message to Serial Monitor
Serial.print("trying WiFi connection ");
Serial.println(WIFISSID);
// wait 1 second (1000 milliseconds)
delay(1000);
// turn off the Green LED
digitalWrite(EMIT_GREEN_LED , LOW);
// wait 2 second (2000 milliseconds)
delay(2000);
}
// if connected ...
// turn on the Green LED
digitalWrite(EMIT_GREEN_LED , HIGH);
// print 'WiFi connection successful' message to Serial Monitor
Serial.println("WiFi connection successful") ;
// print WiFi network settings (inc. IP Address) to Serial Monitor.
Serial.println("IP address: ");
Serial.println(WiFi.localIP());
}
// define function for getting local time as string
// returns a pointer to a string containing the formatted time.
char* get_time_str(void){
// string holding well formatted time.
static char my_time_string[60];
// structure holding the time
// definition is as follow
// struct tm
// {
// int tm_sec; // seconds from 0 to 59
// int tm_min; // minutes from 0 to 59
// int tm_hour; // hours from 0 to 23
// int tm_mday; // day of the month from 1 to 30
// int tm_mon; // month of the year from 0 to 11
// int tm_year; // number of years from 1900
// ... other variables not in the scope of the tutorial..
// };
struct tm my_time;
// read the time into timeinfo structure.
if(!getLocalTime(&my_time)){
Serial.println("Failed to obtain time");
return NULL;
}
snprintf(my_time_string, // string to add the formatted time to.
60, // string length to prevent overflow
"%04d-%02d-%02d %02d:%02d:%02d", // time string format
(my_time.tm_year + 1900),
(my_time.tm_mon + 1),
my_time.tm_mday,
my_time.tm_hour,my_time.tm_min,my_time.tm_sec);
// return the formatted time string
return my_time_string;
}
void setup() {
// Enable the serial terminal
// Useful for debugging messages
Serial.begin(115200);
// configure RED LED GPIO to be OUTPUT
// so that we would control the LED (ON , OFF)
pinMode(EMIT_RED_LED , OUTPUT);
// on the beginning we will turn OFF the RED LED
digitalWrite(EMIT_RED_LED , LOW);
// configure Green LED GPIO to be OUTPUT
// so that we would control the LED (ON , OFF)
pinMode(EMIT_GREEN_LED , OUTPUT);
// on the beginning we will turn OFF the GREEN LED
digitalWrite(EMIT_GREEN_LED , LOW);
// initialize the DHT communication
dht.begin();
// attempt a wifi connection
wifi_connect();
// wait 5 seconds ( 5000 milliseconds ) for network to settle
delay(5000);
// synchronise time with the NTP server.
configTime(gmtOffset_sec, daylightOffset_sec, ntpServer);
// initialize the MQTT client with the broker name and port number
mqttclient.setServer(mqtt_broker, mqtt_port);
// set the reception callback of the MQTT client to be called by the client stack on reception of a new message.
mqttclient.setCallback(mqttclient_callback);
}
void loop() {
// handle the whole MQTT connection
mqttclient_connection();
//Wait for 5 seconds ( 5000 milliseconds )
delay(5000);
// turn the red LED onma
digitalWrite(EMIT_RED_LED , HIGH);
// Print a debugging message on the Serial Terminal indicating the start of the read attempt.
Serial.println("Reading AM2302 ...");
// Reading temperature or humidity takes about 250 milliseconds!
// Sensor readings may also be up to 2 seconds
float humidity = dht.readHumidity();
// Read temperature as Celsius (the default)
float tempC = dht.readTemperature();
// Read temperature as Fahrenheit (isFahrenheit = true)
float tempF = dht.readTemperature(true);
// turn the red LED off indicating the end of the read cycle
digitalWrite(EMIT_RED_LED , LOW);
snprintf(JSONstring,150,
"{\"TimeUTC\":\"%s\",\"TemperatureC\":\"%f\",\"Humidity\":\"%f\"}",
get_time_str(),tempC,humidity);
// Print the read values on the Serial Monitor.
Serial.println(JSONstring);
}We've now completed the hard work ... setting up the MQTT client. Using the client to transmit data in our loop() is now easy!
First, we need to call the mqtt_connection() function contianously in loop to maintain the connection.
mqttclient_connection();We're now ready (finally!) to send some data. We've already created the 'JSONstring' that we're going to send. all that's left is to publish our 'JSONstring' to the 'topic' we're going to create via the broker. The topic is automatically created the first time it is published to. We publish our MQTT message using the mqttclient.publish() function as follows:
mqttclient.publish(deviceUUID, JSONstring);BUT there are lots of reasons the connection to the broker might fails (many of them beyond our control) such as a temporary drop out of the broadband connection into the building, a short-term restriction in network bandwidth, etc.
As a result, in order to build a resilient IoT device, we need to build in some robust error handling. We'll protect the call of the publish function with a check if the MQTT client is really connected, and on such a case we'll attempt to re-connect to the broker on our next call to mqtt_connection() function.
if(mqttclient.connected())
mqttclient.publish(deviceUUID, JSONstring);
else
Serial.println("failed to publish, no connection...");With the new MQTT publishing code added, our complete loop should look like this:
void loop() {
// handle the whole MQTT connection
mqttclient_connection();
//Wait for 5 seconds ( 5000 milliseconds )
delay(5000);
// turn the red LED onma
digitalWrite(EMIT_RED_LED , HIGH);
// Print a debugging message on the Serial Terminal indicating the start of the read attempt.
Serial.println("Reading AM2302 ...");
// Reading temperature or humidity takes about 250 milliseconds!
// Sensor readings may also be up to 2 seconds
float humidity = dht.readHumidity();
// Read temperature as Celsius (the default)
float tempC = dht.readTemperature();
// Read temperature as Fahrenheit (isFahrenheit = true)
float tempF = dht.readTemperature(true);
// turn the red LED off indicating the end of the read cycle
digitalWrite(EMIT_RED_LED , LOW);
snprintf(JSONstring,150,
"{\"TimeUTC\":\"%s\",\"TemperatureC\":\"%f\",\"Humidity\":\"%f\"}",
get_time_str(),tempC,humidity);
// Print the read values on the Serial Monitor.
Serial.println(JSONstring);
// only call the publish function if the client is really connected to the broker
if(mqttclient.connected())
mqttclient.publish(deviceUUID, JSONstring);
else
// if not connected the mqtt_connection function will try to recover shortly on the next loop
Serial.println("failed to publish, no connection...");
}OK, now it's time to compile and upload our Sketch and do some testing!
To help with this, in order to avoid you having to create some application code to connect to the broker, subscribe to your new topic, display the data, etc ... you never are quite sure if any bugs are in your firmware or your subscriber application! ... we've create EMIT 'LIVE' a real-time Environmental Monitoring dashboard for the EMIT project.
We’ve already done the coding and integration to MQTTHQ.com so all you need to do (once your EMIT firmware is uploaded and running) is head over to https://controlbits.com/emit/live/ and paste your deviceUUID into the 'EMIT UUID/Topic name:' box and click the 'View my EMITs data' button.
All being well, you should now see a dashboard showing your MQTT data ... updated every 5-seconds or so!
Notes:
- If nothing appears on the graphs, check the 'Connection Status' box (bottom-right), you may need to refresh your browser a few times to get your browser to connect to the MQTTHQ broker.
- The relay data will be saying 'undefined' ... simply because we've not implemented our relay code yet ... that's our next task: 7. Using the relay for control