As the development of the smart home market, the need for controlling smart devices using human voice grows rapidly. The leading companies all over the world introduced many different solutions in the area of smart home to satisfy their customers’ requirements. Among those solutions, the most popular pattern is to control smart devices via a central smart speaker, which listens to the user’s voice command and responds with actions, such as turning on/off smart lights, adjusting the room temperature through sending remote signals to the AC, and even cooking a nice meal by operating an intelligent robot. However, one thing often happens that breaks this good image is that the customers often find the smart speaker does not support their smart devices. That happens mainly due to one big reason—the communicating protocols used by the smart speaker and the endpoint device are not same.
Currently, there are three protocols mostly used in the smart home industry: Bluetooth mesh, Zigbee, and Z-Wave. In Chinese and the U.S. market, Bluetooth mesh and Zigbee are the mainly used protocols due to their many advantages, such as low power consumption, extendible network size, and strong security.
In the US, the Zigbee is selected by Amazon as the communicating protocol between its smart speaker brand —Amazon Echo -- and the various smart devices in its ecosystem.
While in China, the Bluetooth mesh is the primary protocol that most smart devices company are using. Therefore, the Amazon Echo is not able to control those devices due to the incompatible communicating protocol.
In this project, a solution is proposed to solve this problem.
A Bluetooth mesh network is going to be controlled by an Amazon Echo Plus 2 via cloud-to-cloud approach. The user can turn on and turn off a Bluetooth mesh light by giving a voice command to the Echo Plus.

• Install the Alexa App in you iOS device. Note that the Alexa App is only available for some special regions, for e,g. US, you need to Login the AppStore with a special Apple ID. And below is an Apple ID hosted by APAC regional apps team, you can feel free to use it.

  • UserName: [email protected]
  • Password: xxxxxxxxxxxxx

• Register the Amazon with your own account or use the available account below. For this demonstration, we have registered the account as below.

  • UserName: [email protected]
  • Password: xxxxxxxxxxxxx

• Also you can / should Sign-in the Alexa App with the same account as above.

• If Amazon requests for account verification with email, please sign in the outlook with the account below.

  • UserName: [email protected]
  • Password: xxxxxxxxxxxxx

• Register AWS in https://portal.aws.amazon.com/billing/signup#/start. The AWS is used for holding the lambda function, and credit card information is needed for registering a new account of the AWS.

  • The account information for AWS is not public.
  • UserName: yuancheng@xxxxxx
  • Password: xxxxxxxxxxxxx

Below is the block diagram of this project.

Amazon Echo is a brand of smart speakers developed by Amazon. Echo devices connect to the voice-controlled intelligent personal assistant service Alexa, and the Echo Plus has a built-in Zigbee hub to easily setup and control your compatible smart home devices.

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume - there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service - all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.

The Alexa skill is the bridge that connects the users to AWS IoT could.
After Amazon Echo receiving the user’s voice commands, it will sent the voice record to AWS Alexa server(AI assistant server), where interprets the voice command to control directives. Then, the Alexa server sends the directives to lambda server, a place running the skill code. In the Lambda server, the specific action is executed according to the content of the directives. For example, if the user says “Alexa, turn on my light”, the Alexa will analysis this voice command and sending a JSON formatted file which listed a bunch of information about the light and actions to be executed to lambda, and then lambda will modify the light status from “OFF” to “ON” in the database.

AWS IoT Core is a powerful platform to manage IoT devices remotely. It has a service called Thing Shadow, which is a JSON document recording the information of the devices it manages. In this project, the AWS IoT plays the role of the system database. It records every device’s status in the Thing Shadow document and could be visited by ESP32 via the integrated AWS freeRTOS SDK. Thing Shadows: a JSON document that is used to store and retrieve current state information.

AWS freeRTOS is running on ESP32 board. It connects the local network to the AWS cloud.
ESP32: ESP32 is the gateway in this project. When it runs, it continuously gets the Thing Shadow document from the AWS IoT Core, and then it translates the JSON directives to simple character strings. Through UART communication, it passes the commands to the provisioner in the Bluetooth Mesh network, and the provisioner will give the specific directives to appointed devices.

In this project, a simple Bluetooth network is created that includes two mesh node and of course a provisioner.
The provisioner in the network to provision new devices and receive commands from ESP32 via UART, and also parse the commands from ESP32 and transfer to the mesh command and transmit it to the mesh network for controlling other nodes.
There are total two Bluetooth mesh nodes in the network, a light node and a switch node. Once the user presses the button on the switch node, the light node would receive a BLE mesh message and turn on/off the embedded LED on it.

Also, the switch node is capable to change the online shadow document. When the user give it a long press on left button, the switch node will send a message to provisioner to update the online shadow document.

In this section, the technical details below will be introduced.

• Alexa Smart Home Skill
• Lambda function
• ESP32 freeRTOS application
• Bluetooth Mesh provisioner

sample codes, including lambda function, Alexa smart home skill, ESP32 freeRTOS application, and the codes of the Bluetooth Mesh provisioner (based on EFR32BG13) will also be explained.

However, because of the limited length of this article, if the reader wants to replicate this project, please read the section of Replication the project for the step by step guidance, and access the GitHub page: Alexa Control Bluetooth Mesh Devices and find the mentioned packages accordingly.

Alexa smart home skill interface is designed for controlling smart home devices using Amazon Echo series smart speaker by Amazon. In this project, an Echo plus 2 was utilized to receive the voice command and transmit the voice command to the Alexa server. The skill is held by a lambda function, which means the code is running on the Amazon lambda server. When a user gives the smart speaker a voice command, the command will be firstly analyzed on Alexa server. Then, a JSON format directive will be sent to lambda. There are many different directives. The most important one is the discovery directive, which indicates the speaker to find any available devices and report back to the Alexa server. In this project, the response is also generated by the same lambda function. After receiving the discovery directive, the lambda function will directly send a response message back. Therefore, the device information needs to be programmed in the codes.

if namespace == 'Alexa.Discovery':  
    if name == 'Discover':  
        adr = AlexaResponse(namespace='Alexa.Discovery', name='Discover.Response')  
        #create capability part for the response  
        adr = create_discover_response(adr)               
        return send_response(adr.get())   

The “AlexaResponse()” function is used to generate the response — a JSON document. It uses **kwags parameter to receive whatever the user appointed attributes. Among those attributes, the “namespace” and “name” is necessary to add. Alexa server will use this information to identify what the response is and give a feedback to the user.

#create the discover response   
#first create the capabilities that the endpoints need   
#then add the endpoints to the response entity, and add the capability information to the endpoints  
def create_discover_response(response):  
    #general response  
    capability_alexa = response.create_payload_endpoint_capability()  
    #specific capabilities  
    #power controller--turn on turn off operations  
    capability_alexa_PowerController = response.create_payload_endpoint_capability(  
        interface='Alexa.PowerController',  
        supported=[{'name': 'powerState'}])  
    #create colorcontroller capability  
    capability_alexa_ColorController = response.create_payload_endpoint_capability(  
        interface='Alexa.ColorController',  
        supported=[{'name': 'color'}])  
    #create brightnessController capability  
    capability_alexa_BrightnessController = response.create_payload_endpoint_capability(  
        interface='Alexa.BrightnessController',  
        supported=[{'name': 'brightness'}])  
    #create colortemperature capability  
    capability_alexa_ColorTemperatureController = response.create_payload_endpoint_capability(  
        interface='Alexa.ColorTemperatureController',  
        supported=[{'name': 'colorTemperatureInKelvin'}])  
    #create lock controller capability  
    capability_alexa_lockcontroller = response.create_payload_endpoint_capability(  
        interface='Alexa.LockController',  
        supported=[{'name':'lockState'}]  
    )  
    capability_alexa_endpointHealth = response.create_payload_endpoint_capability(  
        interface='Alexa.EndpointHealth',  
        supported = [{'name':'connectivity'}]  
    )       

The capability and endpoint id compose the virtual representation of a smart device. One device can simultaneously own several different capabilities. For example, a light can support PowerController, ColorController, and ColorTemperatureController capabilities to control the on/off, color, and temperature attributes. After constructing the capabilities of the device, we also need to add other device information and the endpoint id to the response.

response.add_payload_endpoint(  
    friendly_name='Sample Switch',  
    endpoint_id='sample-switch-01',  
    manufactureName = 'silicon labs',  
    display_categories = ["SWITCH"],  
    discription = 'silicon labs product',   
    capabilities=[capability_alexa, capability_alexa_PowerController])  

The "display_categories" item will tell Alexa what type of the device is. Alexa will show the corresponding icon on the Alexa phone application. The discovery response is needed only once. It is similar to the process of registering the device to Alexa server. After this registration, Alexa is connected to the smart devices; the user can see the device on the Alexa application.
Every time Alexa gives a directive, the lambda function will respond a corresponding message to Alexa and execute the directive via changing the status of the device on the IoT core. In the lambda function, every controller interface needs a specific method to handle.

def respond_brightnessControl_dir(request):  
    # Note: This sample always returns a success response for either a request to TurnOff or TurnOn  
    endpoint_id = request['directive']['endpoint']['endpointId']  
    correlation_token = request['directive']['header']['correlationToken']  
    token = request['directive']['endpoint']['scope']['token']  
    name = request['directive']['header']['name']  
    #get the brightness from the directive  
    brightness_value = request['directive']['payload']['brightness']  
    thingshadow_updated = update_thing_shadow(thing_name_id=endpoint_id,brightness=brightness_value)  
      
    #check the operation if successful  
    if not (thingshadow_updated):  
        return AlexaResponse(  
        name='ErrorResponse',  
        payload={'type': 'ENDPOINT_UNREACHABLE', 'message': 'Unable to reach endpoint database.'}).get()  
    #abtr: alexa brightness response  
    abtr = AlexaResponse(  
        coorelation_token = correlation_token,  
        endpoint_id=endpoint_id,  
        token = token)      
    abtr.add_context_property(namespace = 'Alexa.BrightnessController',name = "brightness",value = brightness_value)  

For instance, a brightness controlling directive for light could be executed and responded by the function above. The lambda function will first analyze the directive, extract the information such as the endpoint_id, the correlation_token (which is used in response to Alexa), and the specific actions to be done. Then the lambda function will access to the IoT core, update the corresponding attribute information on the appointed virtual device using update_thing_shadow function. If the updating action is done successfully, lambda will give an acknowledging response to Alexa; else it will give an ErrorResponse.

As mentioned above, the Lambda function is the place to hold Alexa skill. The codes in the lambda function are basically the essence of the Alexa skill. However, to associate a Lambda function and an Alexa smart home skill, several steps need to be done. Please see the section Replication the project for the detailed steps for replicating this project on your end. More information about the lambda function, please refer to the online documentation EN | CN provided by Amazon. The lambda function has multiple servers in different areas of the world. However, some of them don’t support Alexa smart home skills. This project chooses US East(N.Virgina) server to hold the skill. It is worth to mention the rating of the lambda function. The AWS lambda has free tiers for each user, up to 1,000,000 free requests per month. In this project, only requests to lambda function and IoT Core are used, so the free tier is totally enough.

Figure: AWS lambda free tiers

AWS IoT Core is another platform of AWS IoT service. It stores the status information of the remote IoT devices in a special service called Thing Shadow. Briefly speaking, the thing shadow is a JSON document for recording the real-time changes of the device status.

Figure: IoT Core Thing Shadow

The Thing shadow can be modified by both lambda function and local mesh network. In this project, it has been set with four nested layers: state, desired/reported, device name, and attributes. The “state” and “desired/reported” is required for each document uploaded, and once the user uploads a new document, only the items in the new document will be updated. The rest of the items will remain the same value. Also, if the desired/reported sections are not the same value, the document will automatically generate a new section called delta, which records the differences between the desired section and reported section.

Figure: JSON document for thing shadow

ESP32 is the network gateway and responsible for downloading the shadow document, uploading new shadow document, and forwarding directives to the local mesh network. It communicates with the provisioner of the mesh network via UART communication. The JSON document will be first converted to a local string directive and then sent to the provisioner board.
The ESP32 board acquires shadow document from IoT core via MQTT protocol. On the IoT Core, each device has a virtual counterpart. To get the shadow document of a device, the user needs to publish a blank message to the MQTT topic $aws/things/device name/shadow/get. The MQTT server will respond with the specified shadow document. Similar operations such as update the shadow and delete the shadow can also be done in this way. The code in the ESP32 follows a linear structure. There are two ways to get the thing shadow from IoT console. For the first one, it begins with establishing internet connection and MQTT connection, then using the function “AwsIotShadow_Get” to acquire the shadow document continuously in an infinite loop. Once it obtained the shadow, it transfers the text into a more straightforward string directive. The provisioner will also give an acknowledging message with the same format after the directive has been executed. The esp32 will according to the message to update the thing shadow document in the next loop. For the second one, the callback mechanism is used which will notify of different desired and reported Shadow states. The second method is recommended, and you can find all of the source for these two methods from the github.

Figure: ESP32

The Bluetooth mesh network is composed of three parts: A switch node, a light node, and a provisioner node. The provisioner in the network is responsible for establishing the mesh network, authorizing new devices, and receiving messages from outside. The provisioner program obeys the “event-driven” pattern in a big “switch” structure. Every coming event will have an event id and the even data. The application invokes different handler functions to extract information and give responses back according to the event id. Through using the UART interrupt, the provisioner can detect messages come from the esp32 board. Also, when the provisioner receives the event id, which indicates the status changes from mesh network endpoints, it will invoke the UART TX handler to send messages to the ESP32. Moreover, the switch node can control the light node. The provisioner also subscribed the switch node message, which means that it will receive a notification when the switch turns on or turns off the light. The status information of the node will be displayed directly on the Alexa phone application, and the user can also control the devices via the app.

This project uses different protocols to do communications among different parts. The MQTT protocol is utilized for communicating between the ESP32 and AWS IoT Core console, where the thing shadow document is stored; and the Bluetooth mesh is used to organize local devices. Between the ESP32 and the provisioner node in the mesh network, a self-defined simple UART protocol is also used to transmit the devices information and attribute information. Each packet of the UART protocol contains 41 char bytes, and the format is shown below.

Figure: UART packet format

The first byte is used to identify what type of operation is. Currently, the operation supports only change device state; the following bytes are used to determine the device type, the attribute name, and the attribute value. What is worth to pay attention is that the attribute value possibly number or string, therefore, in the code it has to be adjusted using the function “atoi()”

Figure: Operation type definition

Figure: Device type definition

Figure: Attribute type definition

Figure: Light default data definition

In this section, a step-by-step instruction will be provided to the reader to replicate the project.

• 1. SiliconLabs Wireless starter kits x3 (EFR32xG12, EFR32xG13 or EFR32xG21)
• 2. Espressief ESP32-DevKitC or ESP-WROVER-KIT x1
• 3. Wire jumpers x3
• 4. CP210x kit(for debugging)

• 1. Simplicity Studio IDE
• 2. Bluetooth mesh SDK v1.5.0 or above
• 3. Amazon AWS freeRTOS SDK
• 4. Any serial port monitor tool (Tera Term)

The picture below illustrates the block diagram of controlling Physical Device (that is Bluetooth Mesh devices in this project) via Amazon Echo. We will split the procedures as several parts below.

• Create and Configure device within AWS IoT
• Create Alexa Skill
• Lambda function
• Download and set up freeRTOS SDK on ESP32
• Reference

AWS IoT provides secure, bi-directional communication between internet-connected devices such as sensors, embedded microcontrollers and the AWS Cloud. This makes it possible for you to collect telemetry data from multiple devices, and store and analyze the data. You can also create applications that enable your users to control these devices from their phones or tablets. The tutorial below shows you how to create resources required to send, receive, and process MQTT messages from devices using AWS IoT. And the section Download and set up freeRTOS SDK on ESP32 show you how to use an MQTT client to emulate IoT device(s).

If you do not have an AWS account, create one by referring to the knowledge base How do I create and activate a new Amazon Web Services account?.

Log into AWS IoT console, open the services and choose N.Virginia as the server. Because some of the necessary features (IoT Core, Amazon FreeRTOS, etc.) are only supported on this server, so it's recommended to choose this server, and continue the following steps. Navigate to the IoT Core Service, below is the screenshot if you log into the AWS IoT console successfully.

Figure: AWS IoT console

Devices connected to AWS IoT are represented by IoT things in the AWS IoT registry. The registry allows you to keep a record of all of the devices that are registered to your AWS IoT account. Register a thing and name it as esp32_btmesh_bridge on your AWS console: click Manage -> Thing -> Register a thing -> Create -> Create a Single thing -> Create certificate -> Download the certificate and Activate. Note: Carefully preserve the thing certificates downloaded. The keys will be used in the amazon freeRTOS SDK running on the ESP32.

Figure: Create a thing

AWS IoT policies are used to authorize your device to perform AWS IoT operations, such as subscribing or publishing to MQTT topics. Your device presents its certificate when sending messages to AWS IoT. To allow your device to perform AWS IoT operations, you must create an AWS IoT policy and attach it to your device certificate. Go back to console -> secure -> policy -> create. The policy statements define the types of actions that can be performed by a resource, you can just follow the screenshot below to set your statement for this policy.

Figure: Create a policy in this interface

After creating the policy, the UI will similar like below.

Go back to the console, open your thing esp32_btmesh_bridge. Click Security -> Your certificate -> Policies -> Actions -> Attach Policy. And choose the Policy you created just now, and click Attach then.

Figure: Attach Policy

Please don’t forget to check/modify the “Resource” of the policy and make sure it's like below.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:Connect",
      "Resource": "arn:aws:iot:us-east-1:879362360571:*"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Publish",
      "Resource": "arn:aws:iot:us-east-1:879362360571:*"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Subscribe",
      "Resource": "arn:aws:iot:us-east-1:879362360571:*"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Receive",
      "Resource": "arn:aws:iot:us-east-1:879362360571:*"
    }
  ]
}

A device's shadow is a JSON document that is used to store and retrieve current state information for a device. The Device Shadow service maintains a shadow for each device you connect to AWS IoT. You can use the shadow to get and set the state of a device over MQTT or HTTP, regardless of whether the device is connected to the Internet. Each device's shadow is uniquely identified by the name of the corresponding thing.

Click on the Manage -> Thing -> Your Things -> Shadow -> Edit, paste the initial shadow document below. The initial shadow document must have the same desired and reported section, if you want to change the shadow document, remember to change the shadow document template on both the Things board and Alexa skill, otherwise the program won’t run correctly.

Figure: Shadow document

Below is the initial Shadow document utilized in this project.

{
  "desired": {
    "Lights": {
      "device info": "default information",
      "brightness": 55,
      "value": {
        "value": 0,
        "hue": 0,
        "saturation": 1,
        "brightness": 1
      },
      "colorTemperatureInKelvin": 3000,
      "ON_OFF": "OFF",
      "LightID": "sample-light"
    },
    "Switch": {
      "Switch value": "OFF"
    },
    "Lock": {
      "Lock value": "LOCKED"
    },
    "Thing name": "sample-switch-01",
    "color": "RED",
    "sequence": [
      "RED",
      "GREEN",
      "BLUE"
    ]
  },
  "reported": {
    "Lights": {
      "device info": "default information",
      "brightness": 55,
      "value": {
        "value": 0,
        "hue": 0,
        "saturation": 1,
        "brightness": 1
      },
      "colorTemperatureInKelvin": 3000,
      "ON_OFF": "OFF",
      "LightID": "sample-light"
    },
    "Switch": {
      "Switch value": "OFF"
    },
    "Lock": {
      "Lock value": "LOCKED"
    },
    "Thing name": "sample-light",
    "color": "GREEN"
  },
  "delta": {
    "Thing name": "sample-switch-01",
    "color": "RED",
    "sequence": [
      "RED",
      "GREEN",
      "BLUE"
    ]
  }
}

After creating the Things, you can use the test function on AWS IoT console to check if the rule that using the MQTT works or not. Click on the “Interact” option, choose the corresponding topic which represents the operations that you wish to do, subscribe to the topic. Below are the all available MQTT topics for subscribe or publish. For example, you can publish the message to the topic $aws/things/<things Name>/shadow/update to update the thing shadow, and subscribe to the MQTT topics $aws/things/<things Name>/shadow/update/accepted and $aws/things/<things Name>/shadow/update/documents for the accepted messages, also you can monitor the topic $aws/things/<things Name>/shadow/update/rejected for debugging purpose. If the message was rejected, you can get the error code by subscribing the $aws/things/<things Name>/shadow/update/rejected topic. Receiving the MQTT message via the subscribing topic $aws/things/<things Name>/shadow/update/accepted means that the thing shadow works now.

Figure: Topic to subscribe

Figure: Subscribe the topic and publish content you want

You can find an example shadow document here. When you send shadow update messages to the topic $aws/things/<shadow-name>/shadow/update, please make sure it at least contains the state field. You can use the message below directly for replicating this project.

{
  "state": {
    "desired": {
      "Lights": {
        "device info": "default information",
        "brightness": 85,
        "value": {
          "value": 0,
          "hue": 0,
          "saturation": 1,
          "brightness": 1
        },
        "colorTemperatureInKelvin": 3000,
        "ON_OFF": "ON",
        "LightID": "sample-light"
      },
      "Switch": {
        "Switch value": "OFF"
      },
      "Lock": {
        "Lock value": "LOCKED"
      },
      "Thing name": "sample-light"
    },
    "reported": {
      "Lights": {
        "device info": "default information",
        "brightness": 55,
        "value": {
          "value": 0,
          "hue": 0,
          "saturation": 1,
          "brightness": 1
        },
        "colorTemperatureInKelvin": 3000,
        "ON_OFF": "OFF",
        "LightID": "sample-light"
      },
      "Switch": {
        "Switch value": "OFF"
      },
      "Lock": {
        "Lock value": "LOCKED"
      },
      "Thing name": "sample-light"
    }
  }
}

Read the documentations below from Amazon for more information about the AWS IoT Service. Register a Device in the AWS IoT Registry Device Shadow Service for AWS IoT

What is Lambda? AWS Lambda lets you run code without provisioning or managing servers. Alexa sends your code user requests and your code can inspect the request, take any necessary actions (such as looking up information online) and then send back a response. You can write Lambda functions in Node.js, Java, Python, C#, or Go. What is Alexa Smart Home Skill? Alexa provides a set of built-in capabilities, referred to as skills. The smart home skill lets a user control and query cloud-enabled smart home devices such as lights, door locks, cameras, thermostats and smart TVs. The relationship between Skill and Lambda? The code for your smart home skill is hosted as a Lambda function on AWS. For a skill that controls smart home devices such as lights, thermostats, and entertainment devices you can use the Smart Home Skill API. In this case, you develop an AWS Lambda function that running n the Lambda to accept device directives from Alexa. You provide code to handle directives in an AWS Lambda function. Your skill receives requests in the form of device directives to control a particular device. Your code then handles the request appropriately (for example, by turning on the requested light or turning up the volume).

After creating and configuring device within AWS IoT, we need to create Alexa Smart Home Skill.

To configure a new smart home skill, you need an account on the Amazon Developer Console. If you don't already have an account, go to https://developer.amazon.com/alexa/console/ask and create an account. Sign in the Alexa developer console with your amazon account.

Choose Create skill->Smart Home->Enter the skill name->Create. Once you created the Smart Home Skill, Your Skill ID will be assigned that will used by Lambda function.

Jump to the next section to Add a Lambda Function and then back to here.

You must provide the ARN for your Lambda function in the skill configuration in the developer console. Navigate back to your skill in the Developer Console. Under 2. Smart Home service endpoint, in the Default endpoint box, provide the ARN number from the Lambda function you created and click Save. Therefore, the skill is associated with the lambda function.

Figure: Lambda function adding Alexa smart home trigger

Question: What is Account Linking? Answer: Account linking enables your skill to connect the user's identity with their identity in a different system. All smart home, video, and meetings skills must connect the identity of the Alexa user with an identity in the service provider's system. This is known as account linking, because the goal is to create a link between the Alexa user and the service provider. For more information about the Account Linking, please see the section Further Reading 2

After you created the skill, the first step is to setup the account linking. Smart Home skills require that a user completes account linking during skill enablement. This step asks customers to associate their device cloud account with your smart home skill. You will need an OAuth provider in order to implement this process. If you don't already have an OAuth provider, you can use Login with Amazon (LWA).

Setup the LWA: For a skill that includes the smart home, video, baby activity, or meetings model, the account linking is required. Account linking must be configured with authorization code grant. https://developer.amazon.com/docs/devconsole/build-your-skill.html#account-linking

• i. Connect to https://developer.amazon.com/login.html and authenticate with your Amazon credentials.
• ii. Click "Login with Amazon"
• iii. Click on “Create a New Security Profile”

• iv. Fill in all three required fields to create your security profile and click “Save”. For the “Consent Privacy Notice URL”, please fill it with your own privacy notice URL. If there is no privacy notice, you can just fill it with https://www.privacypolicies.com/blog/gdpr-consent-examples/.

• v. Before you complete this step, be sure to click on the link named “Show Client ID and Client Secret” and save these values to a secure location so they're easily available later. You’ll need these values later in a next step.

• i. Go back to https://developer.amazon.com/home.html and sign in as needed
• ii. Go to Alexa > Alexa Skills Kit > the Alexa skill you created earlier
• iii. In the Configuration tab:
• iv. Lambda ARN default = enter your Lambda ARN noted in the next section.
• v. Authorization URI = https://www.amazon.com/ap/oa
• vi. Access Token URI = https://api.amazon.com/auth/o2/token
• vii. Client ID = your client ID from LWA noted in a previous step
• viii. Client Secret: your client secret from LWA noted in a previous step
• ix. Client Authentication Scheme: HTTP Basic (Recommended)
• x. Scope: profile (click Add Scope first to edit it)
• xi. Click Save
• xii. Provide the Redirect URL's (as illustrated in the following screenshot) to LWA:
• xiii. The Configuration page for your Skill lists several Redirect URL's. Open the LWA security profile you created earlier and visit the Web Settings dialog. Provide each of the Redirect URL values from your Skill in the “Allowed Return URLs” field.

After mutual association, the second step would be account linking. On the Alexa console, click on the account linking option, input the authorization URI and the access token URI as shown below.

Figure: Account linking

Amazon console also supports the skill-testing. However, to test a SmartHome or Video skill, you need first enable the skill with the Alexa companion app (cannot finish it with webpage).

And then you can try to discovery new devices with the App. After finishing the discoverying process, it will list all of the devices it discovered.

Then you can control all of the discovered devices with the Alexa App, also you can test them on the Alexa developer Console. The developer can choose either typing or using JSON formatted directive to test the Alexa skill. It simulates a real echo plus speaker to send the directives to the server where holding the Alexa skill. For example, if the user type or say “turn off light”, the Alexa server will send a turn-off directive to the lambda(the skill holder server).

Figure: Skill-testing

The code for your smart home skill is hosted as a Lambda function on AWS. AWS Lambda is a service that lets you run code in the cloud without managing servers. Alexa sends your skill requests and your code inspects the request, takes any necessary actions such as communicating with the device cloud for that customer, and then sends back a response.

1. Go to https://console.aws.amazon.com/console/home and sign in
2. Switch the AWS server to “US East (N.Virginia)” on the top right of the page, because some of the features are only supported on this server. Below is the recommended service for different regions.

• N.Virginia for English (US) or English (CA) skills
• EU (Ireland) region for English (UK), English (IN), German or French (FR) skills
• US West (Oregon) for Japanese and English (AU) skills.

3. Go to Services > Compute > Lambda
4. Click on Create Function
5. Click on “Author from scratch”
6. Configure your Lambda function

• Name = alexaBtmeshBridgeLambda (or whatever you want)
• Runtime = Python 3.6 (You can choose any language used to write your function, in this demo, we are using Python 3.6)
• Permission = “Create a new role with basic Lambda permissions”
• Click Create Function

7. Click Triggers -> Add Trigger and select Alexa Smart Home

• Application Id = Skill ID of your test skill that your created above.
• Enable trigger = checked
• Click Submit

8. Click the icon of the lambda function for Configuration

Regarding to the lambda function, we take the Alexa Smart Home skill sample released by Amazon as starting point, you can access it from the repository skill-sample-python-smarthome-switch. And also you can just reuse the lambda function we have implemented for this project, the zip package alexa-control-btmesh-lambda.zip is included in the sub directory and also you can get it from the github.

• Runtime = Python 3.7
• Code entry type = select the "Upload a .ZIP file" (the Alexa skill package)
• Click on Upload and find the alexa-control-btmesh-lambda.zip you created earlier. The Things name is esp32_btmesh_bridge in the lambda_function.py, update it according to the real Thing name you created before.
• Handler = self_defined_lambda.lambda_handler
• Click Next
• Click Save
• After uploading the .ZIP file, please make sure that the lambda_function.py in the root folder.
• On the top right corner, note the Lambda ARN.
• Lambda ARN: Copy the ARN and it will be used in the Alexa Skill section as mentioned above.

And also please pay attention to the "Existing role" of the Lambda function, we will configure the role in the next section Configure the IAM Role for Lambda.

After uploading the ZIP package successfully, please make sure that the directory structure is similar as below, and the lambda function file name should be lambda_function.py

IAM (AWS Identity and Access Management) is a web service that helps you securely control access to AWS resources. You use IAM to control who is authenticated (signed in) and authorized (has permissions) to use resources. Please see the user guide for how to set up your account and permission with IAM.

To grant the role with the following policy, the role was created automatically during creating the Lambda function which is alexaBtmeshBridgeLambda-role-lft8jt47 here.

Attach the following IAM policies to the role used by the Lambda function:

• AmazonFreeRTOSFullAccess
• AWSIoTFullAccess
• AWSIoTDataAccess
• AWSLambdaFullAccess

Now navigate back to your skill in the Developer Console.

If you want to perform a quick test of the Discovery request in the Lambda code editor, you can click on Test. The Configure test event page appears. Leave Create new test event selected. For Event template, leave the default Hello World. In the Event name, enter discovery and replace the entire contents in the editor with the following code.

{
    "directive": {
        "header": {
            "namespace": "Alexa.Discovery",
            "name": "Discover",
            "payloadVersion": "3",
            "messageId": "1bd5d003-31b9-476f-ad03-71d471922820"
        },
        "payload": {
            "scope": {
                "type": "BearerToken",
                "token": "access-token-from-skill"
            }
        }
    }
}

Do the Test, if successful, the Execution Result should be similar to the following:

{
  "event": {
    "header": {
      "namespace": "Alexa.Discovery",
      "name": "Discover.Response",
      "messageId": "00c40bcd-75e5-406e-a4d2-de476dbc7992",
      "payloadVersion": "3",
      "correlation_token": "INVALID"
    },
    "payload": {
      "endpoints": [
        {
          "capabilities": [
            {
              "type": "AlexaInterface",
              "interface": "Alexa",
              "version": "3"
            },
            {
              "type": "AlexaInterface",
              "interface": "Alexa.PowerController",
              "version": "3",
              "properties": {
                "supported": [
                  {
                    "name": "powerState"
                  }
                ],
                "proactivelyReported": true,
                "retrievable": true
              }
            }
          ],
          "description": "Sample Endpoint Description",
          "displayCategories": [
            "SWITCH"
          ],
          "endpointId": "sample-switch-01",
          "friendlyName": "Sample Switch",
          "manufacturerName": "Sample Manufacturer"
        },
      ]
    }
  }
}

Steps to Build a Smart Home Skill

Understand Account Linking

Go to GitHub page to download the SDK or clone it by using git clone https://github.com/aws/amazon-freertos.git, and also you can clone the forked amazon freeRTOS SDK that we are using for this project.

Setup the SDK with correct Toolchains and establish the serial connection to your ESP32 board. The very detailed official instruction can be found HERE EN | CN.

Read the page here EN for how to get start with CMake on windows. ESP-IDF is the official development framework for the ESP32 chip. Please note that due to some compatibility issue, the version 3.1.5 of ESP-IDF (the version used by Amazon FreeRTOS) does not support the latest version of the ESP32 compiler. You must use the compiler that is compatible with version 3.1.5 of the ESP-IDF. The compatible compiler is provided within the installation package esp-idf-tools-setup-1.1.exe, please get the installation package from the link above.

The installer will automatically install the ESP32 Xtensa gcc toolchain, Ninja build tool, and a configuration tool called mconf-idf. The installer can also download and run installers for CMake and Python 2.7 if these are not already installed on the computer. However, the CMake included in the installer is an elder version, for CMake installing method, please read the section Install CMake below.

After setup has finished installing ESP-IDF tools, the GUI now should looks like this.

Click the Finish button. By default, the ESP-IDF installer updates the Windows Path environment variable automatically so all of these tools can be run from anywhere.

If you didn't see these path included in the System Environment, you will need to configure the environment where you are using ESP-IDF with the correct paths by yourself.

Install the CMake (Cross-Platform Makefile Generator). The CMake build system is required to build the Amazon FreeRTOS demo and test applications for this device. Amazon FreeRTOS supports CMake versions 3.13 or above, and you can get download the latest version of CMake from CMake.org.

Form you native build system, it can be GNU Make or Ninja. Ninja is recommended by Amazon since it is faster than Make and also provides native support to all desktop operating systems. And we also use the Ninja in this project.

Use the following CMake command to generate the build files, and then use ninja to build the application. cmake -DVENDOR=espressif -DBOARD=esp32_devkitc -DCOMPILER=xtensa-esp32 -GNinja -S . -B your-build-directory

Change directories to the build directory your-build-directory you just specified above, i.e, build. Invoke Ninja to build the application. ninja

Establish serial connection between your host machine and the ESP32 kit. You must install CP210x USB to UART Bridge VCP drivers. You can download these drivers from Silicon Labs.

Use Espressif's IDF utility ( amazon-freertos/vendors/espressif/esp-idf/tools/idf.py ) to flash your board, run the application, and see logs. To erase the board's flash, go to the directory and use the following command:
python ./vendors/espressif/esp-idf/tools/idf.py erase_flash -p COM17 -B build

To flash the application binary to your board, use the IDF script to flash your board: python ./vendors/espressif/esp-idf/tools/idf.py flash -p COM17 -B build

To monitor: python ./vendors/espressif/esp-idf/tools/idf.py monitor -p COM17 -B build

You can combine these commands. For example, suppose the com port of your ESP32 kit is COM5: python ./vendors/espressif/esp-idf/tools/idf.py erase_flash flash monitor -p COM17 -B build

Make sure that you can build the default demo, and flash it to the ESP32 kit, and also can get debug log output on the console.

Please carefully read the documentation page on how to setup the AWS freeRTOS SDK on your computer. The chinese version documentation is available here.

In the file: …/vendors/espressif/boards/esp32/aws_demos/config_files/aws_demo_config.h, enable the shadow demo by #define CONFIG_SHADOW_DEMO_ENABLED. If you are using the amazon freeRTOS sdk cloned from Amazon repo, please open the SDK folder, in the folder …/demos/shadow, replace the C file aws_iot_demo_shadow.c with the provided aws_iot_demo_shadow.c file. Also, add the file aws_iot_shadow_blem.h to the folder.

The WiFi settings can be done in the file …/demos/include/aws_clientcrediential.h Set clientcredentialMQTT_BROKER_ENDPOINT as your AWS IoT endpoint. You can find it from the Things' interact information.

Set clientcredentialIOT_THING_NAME to the unique name of your IoT Thing. Set clientcredentialWIFI_SSID to your network name. Set clientcredentialWIFI_PASSWORD as your password.

Paste the certificate you acquired when you create the “esp32_btmesh_hub” thing on AWS IoT console to the file …/demos/includes/aws_clientcrediential_keys.h.

Alternatively, you can use the quick setup which instructed on this page under Configure the Amazon FreeRTOS Demo Applications section.

The esp32 will parse and forward the received MQTT message to the bluetooth provisioner (EFR32BG13) via uart port, so you need to configure the serial port for ESP32. GPIO16 and GPIO17 of ESP32 are configured as Tx and Rx pin in this project.

#define ECHO_TEST_TXD (GPIO_NUM_16)
#define ECHO_TEST_RXD (GPIO_NUM_17)

a. Download the provisioner project. b. Open the project in simplicity studio, build, flash to BG13 board

a. The mesh light and switch are using the demo applications in the simplicity studio Bluetooth mesh SDK v1.5.0, you can use other Bluetooth mesh light device in this project. b. The mesh switch application is also the demo program in simplicity studio, it has three functions: adjusting the lightness of the light, adjusting the temperature, turning on/off the light. A long press will turn on/off the light, a short press will adjust the lightness, and a normal press will change the temperature. Currently, the provisioner only subscribes the on/off function, when the user give it a short press or long press, the provisioner will report the light’s on/off status to IoT console(while the lightness is not 0, the light’s on/off state will be “ON”). c. Sensor/monitor program. The sensor/monitor program is only available after the version 1.5.0 of the Bluetooth mesh SDK. Currently, the IoT console doesn’t monitor the state of the sensor. Once the sensor and the monitor are provisioned, they are functioning as local devices.

Bluetooth mesh network operation a. Flash the provisioner program to the BG13 board. b. Hold left button and reset button to reset the board c. Flash the switch program and light program into board d. Open a COM tool and connect it to the BG13 board, when seeing the prompt to provision the switch board and light board, press left button to provision.

As the Alexa App has discovered all of the device we defined in the ESP32_btmesh_bridge Things, and you can control the bluetooth mesh node by using the Alexa App now.

Port EFR32 on Amazon FreeRTOS. And then we can interface to AWS IoT by using the combination of EFR32+WGM160P.

This project uses multiple protocols and Amazon AWS services. It breaks the limitation between the Zigbee embedded smart speaker and Bluetooth mesh embedded smart devices and realizes voice control. Although it is just a brief demonstration, it reveals the potential of multi-platform cooperation and smart automation.