Data in the Air: A Simple Guide to Using MQTT Between an ESP32 and a Laptop - ECE-180D-WS-2023/Knowledge-Base-Wiki GitHub Wiki

Introduction

A microcontroller is an integrated circuit designed to act as the brain of an embedded system. Currently, there are several types of microcontroller in the market, but the main focus of this article is a microcontroller called ESP32. The ESP32 is being extensively used by roboticists, students, and engineers in this era due to its ability to connect and exchange data with other sensors or software over the internet. This structure of connection is what is called the Internet of Things (IoT). This process is very important but intricate to develop, and this article covers a basic explanation, the reason to use MQTT. and tutorial when it comes to establishing connections between IoT devices such as an ESP32 and a laptop.

Wired vs. Wireless Communication

So, how do you send data from a microcontroller to a laptop?

Establishing a communication between these two can occur in two ways, wired and wirelessly. Most of the time, a wired connection in the form of USB (Universal Serial Bus) is used to send or receive data from one device to another. One good example is the common way of uploading a code from an electronic prototyping platform like Arduino. Wired connections are regularly supported by most devices. When a microcontroller is bought, one can instantly hook it up with a laptop using USB and begin controlling or getting data from it.

The downside is that one always needs a wire to do this. This is suitable if the proximity between the devices does not matter. However, when one decides to use an ESP32 microcontroller with a camera sensor to implement a color-detecting flying robot while sending the data real-time to a laptop for processing, a wired connection will not work. This is when wireless communication is used.

Wireless communications between devices are mostly done through the use of the Wi-Fi or the Internet. This is the main backbone of the Internet of Things and how embedded systems like smartphones or wearable devices can send data to another device of the same kind. In this type of communication, wires do not matter, making proximity of devices longer. However, since this type of connection uses the Internet, it is more susceptible to interference, resulting in lower latency than what a wired connection can offer.

There are several ways to establish wireless communications, but this article will mainly focus on a simple messaging protocol called MQTT which stands for Message Queuing Telemetry Transport. MQTT is a standard messaging protocol of IoT (Internet of Things) devices (MQTT.org, 2022). The said protocol is utilized in machine-to-machine communication using a simple publish/subscribe communication pattern (Bernstein et al., 2021).

Why MQTT?

MQTT is a lightweight protocol and costs less bandwidth. A widely used protocol “HTTP” is a request-response protocol in the client-server model. While the size of HTTP protocol headers is about 8000 Bytes, the MQTT protocol header size is only 2 Bytes. This means HTTP packets are larger thousands of times compared to MQTT packets. As we know, the number of IoT devices is growing rapidly and some researchers predict that it will exceed 50 billion in the next years. Using lightweight protocol as MQTT can prevent overload on the network as IoT devices need to send and receive data frequently. For the device itself, another advantage of using MQTT is that it will cost less memory and less power as it is more suitable for small IoT devices. Researchers compare MQTT and HTTP in battery-powered devices. As a result, MQTT-based communication consumed 93% less energy than HTTP-based communication for Wi-Fi and 82% less energy for Bluetooth. In a benchmark test run by HiveMQ, a MQTT broker, MQTT is found to be capable of managing over 10 million concurrent connections and processing more than 1 million messages per second. This demonstrates the protocol's capacity to scale and provide high message throughput support for a significant number of IoT devices.

For some applications, IoT connectivity may periodically be unstable due to the mobility and battery power of particular IoT devices. Making ensuring the recipient receives the info is important in some circumstances but not in others. Reliability is important in complicated circumstances like Industrial IoT, as data loss or duplication can result in system failures. MQTT, which has its roots in the oil sector, was created to offer the highest level of reliability even in difficult network circumstances. By including Quality of Service (QoS) flags, which ensure message delivery, the MQTT protocol improves reliability. Three QoS levels exist: QoS 0, QoS 1, QoS 2. MQTT utilizes QoS 0 by default, which is appropriate for stable network settings when minor data loss is manageable. For instance, ambient sensors often communicate data in smart home projects, thus losing a few bits of information won't have a big impact. The fastest speed and least amount of network traffic are provided by QoS 0, also known as "Fire and Forget.” When message delivery is essential and duplicate messages are acceptable, QoS 1, a higher level, assures that messages are received by data consumers at least once. For crucial situations where message delivery is essential and duplicate messages are not acceptable, QoS 2 offers the best messaging quality. QoS 2 is slower than the other levels but it provides the most trustworthy communication.

The vast availability of open-source libraries and tools for MQTT across numerous programming languages illustrates its popularity and ease of adoption among developers, even though it is difficult to give quantitative evidence on implementation and maintenance simplicity. Additionally, compared to more sophisticated protocols like AMQP, MQTT's straightforward and simple protocol design makes it simpler to deploy and maintain.

The advantages of MQTT in terms of bandwidth use, power consumption, scalability, and ease of installation are highlighted by these quantitative data points, making it a solid option for IoT applications.

Publish/Subscribe Communication Pattern of MQTT

In the above diagram, three components can be seen: publisher, subject, and subscriber. Using these three components, the flow of MQTT is described as follows:

  1. Publishers create their own unique and specific subjects, also known as topics. These subjects will serve as the label for their data.
  2. Publishers then send their data labeled with a subject/topic into an MQTT broker which is an intermediary between a publisher and a subscriber.
  3. The MQTT broker sends the data to the specific subscriber that is subscribed to the subject/topic created by the publisher. Subscriber receives data from publisher.

This is a high-level description of how MQTT works. The major component of MQTT is the broker which allows the publisher and subscriber to communicate to each other. The MQTT broker takes the messages published by the publisher, filters these messages according to their subjects/topics, and then proceeds to send them to the subscribers (Chaurasia, 2020).

The implementation of MQTT is complex, but there are several open-source libraries across the Internet that allow people to use MQTT easily, making it uncomplicated to incorporate the said protocol into devices. The basic usage of MQTT using open-source libraries will be covered in the next section.

Tutorial

In this tutorial, we will enable an ESP32 microcontroller to send data to a laptop using MQTT. The data that will be sent for this tutorial is a simple integer, but these data can easily be replaced by sensor or telemetry values.

There are two MQTT broker that can be used in these tutorial. They are both open-source – test.mosquitto.org or mqtt.eclipseprojects.io.

At the end of this tutorial, one will be able to establish an MQTT connection between an ESP32 microcontroller and a Laptop.

Materials Needed (Hardware and Software)

  1. Laptop
  2. ESP32 Microcontroller
  3. Python3 Installed in your Laptop
  4. Arduino IDE
  5. WiFi with known SSID and Password (not 5G)
  6. Directory in your Laptop where you want to store the contents of this tutorial call it MQTT_Basic

Setting up the Laptop

  1. Install and download Miniconda in accordance with your Operating System.
  2. Create an environment in your Laptop and install the MQTT library for Python. This can be done by executing the following commands (macOS):
$ cd [path to MQTT_Basic directory]
$ conda update conda
$ conda create -n yourenvname python
$ conda activate yourenvname
$ pip install paho-mqtt
$ conda deactivate

Setting up the Arduino IDE

  1. Add ESP32 boards in your Arduino. Follow this tutorial.
  2. Install the PubSubClient by Nick O’Leary

Understanding the Code

The MQTT code tutorial will be divided into two sections, the ESP32 side and the laptop side. In this tutorial, the ESP32 will be the publisher and the laptop will be the subscriber. Below are the instructions and explanations of some important code snippets. The full code has comments inside that makes it easier to understand.

Links to full code: ESP32 as the publisher code Laptop as the subscriber Code Make sure to open these two links while reading the sections below.

ESP32 Side (Arduino C/C++)

Enter your WiFi credentials (lines 7 to 8)

const char *ssid = "";      // Enter your WiFi name
const char *password = "";  // Enter WiFi password

Note: Some WiFis my not allow your ESP32 to connect to the internet. Common WiFis that incur this issues are public WiFis and ones that are 5G. Using a non-5G private WiFi or a mobile hotspot works best.

Choose your MQTT Broker by uncommenting one of the two lines below (lines 13 and 14)

// const char *mqtt_broker = "test.mosquitto.org";
const char *mqtt_broker = "mqtt.eclipseprojects.io";

Note: If one broker does not connect, try the other.

Enter your topic/subject (line 16)

const char *topic = "test/topic";   // Enter your topic name here

Note: Choose a topic as unique as possible, and by convention, label topics with front slash: this/is/a/unique/topic

Publishing Data

Codes in the setup() section of the code are mainly for setting up the baudrate, connecting to WiFi, and connecting to the MQTT broker. The main codes for publishing to a subscriber are inside the loop() function.

In the code snippet below, the function client.publish() sends a data which is the integer 1 to any subsriber subscribed to the topic specified as the function's argument. The function client.loop() manages the outgoing and incoming data/messages buffer of the MQTT. In short, it maintains network traffic flow with the broker.

client.publish(topic, 1);
client.loop();

Upload the full code

You can see the instructions for uploading a sketch/code to your ESP32 here.

Laptop Side (Python)

Choose the same MQTT Broker and topic that you used in the ESP32 code (lines 12 and 41).

client.subscribe("test/topic", qos=1)
...
client.connect_async('mqtt.eclipseprojects.io')

Receiving Data

The function on_message() is used as the default message callback. This means that we receive any data from the topic we subscribed using this function. The most important parameter here that we use is the object message. It contains several flags in which you can access. For instance, we use message.payload to get the data corresponding to the topic that we are subscribed to and message.topic to get the topic in which received the data with

def on_message(client, userdata, message):
 print('Received message: "' + str(message.payload) + '" on topic "' +
       message.topic + '" with QoS ' + str(message.qos))

Running Code

After making all the changes above and making sure that your laptop is connected to the internet. Go to the directory of your Python code and activate conda environment that you created during the setup. This can be done by executing the following commands in your terminal (macOS):

$ cd [path to MQTT_Basic directory]
$ conda activate yourenvname
$ python3 mqtt_laptop_subscriber.py

Expected Output

Once each respective code is uploaded or executed. The integer 1 should be received on the laptop side. An example output on terminal can be seen below:

Received message 1 on topic test/topic with QoS 1  
Received message 1 on topic test/topic with QoS 1  
Received message 1 on topic test/topic with QoS 1

Conclusion

This Wiki Article showed how to send a simple integer from an IoT device like the ESP32 to a laptop using a message protocol called MQTT. The codes shown in this tutorial can be extended in such a way that sensor/telemetry values like those outputted by IMU sensors can be sent from the ESP32 to the laptop for processing. At the same time, the tutorial above can also be extended to send data from a laptop to an ESP32 microcontroller.

Ultimately, there are two ways when it comes to establishing a communication between two devices like an ESP32 and a laptop – wired and wireless. The choice between the two modes depends on the application, but the main deciding factors to consider are proximity and latency. However, as the world maximizes the usage of the Internet, wireless communication is being preferred than wired despite of the factors mentioned above. Between the choice of wireless communication, MQTT is being used in some specific fields. This is the reason why it is important to learn the basics of implementing this type of connection, and you just started by reading and following this article.

References

Bernstein, C., Brush, K., & Gillis, A. S. (2021, January 27). What is MQTT and how does it work? IoT Agenda. Retrieved February 8, 2023, from https://www.techtarget.com/iotagenda/definition/MQTT-MQ-Telemetry-Transport Hertz, Daniel. “Should I Use a Wired or Wireless Connection in My Project?: Custom.” Maker Pro, Maker Pro, 31 Jan. 2019, https://maker.pro/custom/tutorial/should-i-use-a-wired-or-wireless-connection-in-my-project. MQTT.org. (2022). The standard for IOT messaging. MQTT. Retrieved February 8, 2023, from https://mqtt.org/ Rothuis, A. (2018, October 7). Messaging Pattern: Publish-Subscribe. A. ROTHUIS. Retrieved February 8, 2023, from https://www.arothuis.nl/posts/messaging-pub-sub/ Chaurasia, A. (2020, November 15). Wired communication protocols in IOT - Pianalytix - machine learning. Pianalytix. Retrieved February 10, 2023, from https://pianalytix.com/wired-communication-protocols-in-iot/ “What Is the Internet of Things (IOT)?” What Is the Internet of Things (IoT)?, https://www.oracle.com/internet-of-things/what-is-iot/. "7 Advantages of MQTT protocol for IoT Devices",https://medium.com/@esperso/7-benefits-of-mqtt-protocol-for-iot-e463f6a97100.