Introduction

In recent years, the term "edge" has being coming up more and more in discussions about the future directions of machine learning ML, and is often talked about in connection with the Internet of Things (IoT).

At the start of this, it makes sense to spend some time discussing some key terms, a number of which overlap each other. When discussing machine learning at the edge, one may hear terms such as "edge AI", or "edge ML". Other common terms would be "embedded machine learning", "embedded ML", "embedded AI", and a popular term: "tiny ML". Essentially, all these terms are interchangeable, and while there may be some differences depending on context, a person can use any of these terms to discuss the same topic.

Embedded Systems

Embedded is quite a common term in the field of electronic engineering. An embedded device, or an embedded system is a computer that controls the electronics of many of today's modern devices. Embedded systems can be found in everything from mobile phones to modern cars to satellites which orbit the earth. These embedded systems are able to run software, which will will control the functions and ability of the system.

Embedded systems are in more places that one may imagine, or it may be more accurate to say that there are more embedded chips in a single device than one may imagine. In 2020, more than 28 billion microcontrollers were shipped globally, and the trend is predicted to grow, with a focus on automation and AI devices [1]. Given how common these devices are, it is quite obvious that researching how best to transfer ML models to these systems is an important step.

The Edge and the IoT

The term 'edge' may still seem slightly unusual. How does edge relate to ML? The edge tends to be something that is located far away.

When discussing the internet, computers, or IT systems, most people will have an image of the PC they have at home, or the computer they use in work. But there are more devices connected to the internet than your PC. In fact, it is estimated that as of 2021, there were 12.2 billion active IoT connections [2]. These IoT devices cover almost any aspect of our lives that one cares to think about, everything from: smart watches, smart kitchen appliances, baby monitors connected to the internet, allowing parents to check in from anywhere in the world, shipping containers, industrial sensors used to monitor the health of machinery. The list goes on.

On the face of it, all these devices are connected to the "internet". But most people tend to think of the internet as something "that comes out of the box in the corner" (i.e. the router) (direct quote from my mother, Irene). How are all these billions of devices connecting to networks and communication? They have connected to servers, and these are the serves which are often referred to as the "cloud".

All of these devices are connected to a network, they are taking readings from their sensors, and sending that information across this network, to a location where it can be stored and processed. From this perspective, these devices sit at the 'edge' of the network, and this is where the term edge devices comes from.

Edge AI

For a lot of time, IoT devices were seen as ways to collect data via the sensors they had onboard. They would collect the data, and then transmit it back to a hub for processing. This approach is very expensive in a number of ways. First, it costs a lot of money to transmit large amounts of data, due mainly to the connectivity and storage costs, there is also the issue that for any battery powered IoT devices, transmitting data is an extremely power hungry task. Second, it is also expensive in human time too, because people will need to evaluate the data and process it, potentially making some decision based on that data analysis.

So, sending information back to a central location to be processed, and then return the result of that processing can take time. Not a lot of time, in terms of how we humans work, but in terms of the timescales a computer works it, the lag in sending your information and getting the response can long. Need to get more details in timings. This lag characteristic, more correctly called latency can be detrimental when the system requires a fast response. For example, it is not difficult to imagine a situation where a self-driving car needs to make a decision on whether or not the 'object' it has detected is a person crossing the road in front of it, and it needs to apply the brakes in an emergency stop. It would not do to have a system that needed to send the data collected off to a server, which had to process it so it could determine if the object was a human, and then send that results back to the computer on the vehicle, which then needs to react to that. By the time all that took place, it may be too late. And let's not even get into what could happen in a situation where there was a momentary loss of connection to the network!

Issues like this has pushed the need to remove that latency. This could be achieved by having faster connections and by increasing the compute speed on the server side. Another way is to remove the need for the server side to do the processing. In this situation, the system that collects the data also processes the data to get a result. This is where we enter the world of Edge AI.

Sources

[1] https://www.businesswire.com/news/home/20211013005793/en/Global-Microcontroller-Market-Size-Share-Trends-Analysis-Report-2021-2028---ResearchAndMarkets.com
[2] https://iot-analytics.com/number-connected-iot-devices
[3] AI at the Edge (book): https://learning.oreilly.com/library/view/ai-at-the/9781098120191/