1.5. Ultrasonic Sensor Funnel tutorial - soracom/handson GitHub Wiki

Preparation

What is SORACOM Funnel

SORACOM Funnel ("Funnel") is a cloud resource adapter that sends data directly from devices to specified cloud services. Simply specifying the supported cloud services and the resources connected to the services using Funnel enables the input of data into the specified resources.

Features of Funnel

A cloud resource adapter specialized for cloud services

Funnel sends data directly from devices to cloud services. While it is similar to SORACOM Beam in hits mechanism, Funnel is a data sending service specialized for cloud services, providing "adapter" connection functionality for varied cloud services.

Conventionally, sending data from IoT devices to cloud services has required installation of an SDK on the IoT devices, or has required customers to set up servers to relay the data to cloud services. In addition, it has been necessary to keep key data, such as passwords for accessing cloud services, on the devices, posing security concerns.

Funnel enables the input of data into specified resources through the simple specification of supported cloud services and the resources to which those cloud services connect. Funnel eliminates the need to store passwords for cloud service access on devices, and allows customers to make speedy use of cloud services with minimal hassle.

As a Funnel adapter, it corresponds to the following.The Funnel adapter works the followings.

  • AWS IoT (REST)
  • Amazon Kinesis
  • Amazon Kinesis Firehose
  • Microsoft Azure Event Hubs
  • Google Cloud Pub/Sub

Funnel Diagram

Funnel is compatible with the HTTP, TCP, and UDP protocols. Devices are able to input data into cloud services through simple implementation of compatibility with these protocols.

How to use Funnel

To use Funnel, you need to configure your SIM group to set which cloud service and credentials to use. In this tutorial, we will use AWS IoT as the traget service, but the fundamentals are same for other cloud-provider/services.

Prepare credentials

Access IAM Management Console and click on "Users" in the left menu, then click on "Add user".

Specify user name (i.e. Funnel) and check "Programmatic access" in Access type, then click "Next: Permissions".

IAM create user: Details

Click on "Attach existing policies directly" and input "iotdata" in search box, then check "AWSIoTDataAccess" permission, then click "Next: Review", click on "Create user" in next screen.

IAM create user: Permissions

You can copy "Access key ID" and "Secret access key" on the screen to some file or just click "Download .csv" to download it to local disk.

Prepare Custom endpoint

If you have never accessed AWS IoT, your dedicated endpoint might not be generated. Open AWS IoT console and change region if you needed. Click on "Test" in the left menu. Then you might see a notification like below (if not, your endpoint is already ready to use).

AWS IoT: provisioning

Click on "Setting" (you don't need to wait for provisioning) in the left/bottom menu, you can check your "Custom endpoint " FQDN. We will use the FQDN, so copy it to some where and go to next step.

Configure SIM group

Let's get back to SORACOM User Console and click on the group name of the SIM to use. Click "SORACOM Funnel" and turn on the toggle switch. Click on "Service" and choose "AWS IoT". Then you will see fields like below.

Funnel: AWS IoT

Set Destination

Change "Destination" to specify which endopoint / topic to use.

i.e.)

  • endpoint: abc1def2ghi3.iot.eu-west-2.amazonaws.com
  • topic: sensor

-> https://abc1def2ghi3.iot.eu-west-2.amazonaws.com/sensor

Register a credentials

Click on "+" button in the right to open the dialog. Fill in required fields and click "Register".

  • "Credentials set ID" is the unique ID of the credentials and could be anything
  • "Description" is also free text field
  • Copy/Paste AWS Access Key ID and AWS Secret Access Key from the .csv file.

Funnel: Credentials

  1. Select "Credentials set" you just created and click on "Save".

You are ready to send data to AWS IoT from your device!

Send sensor data to AWS IoT via Funnel

Download a python script and run it.

You need to install python-requests package ( it should be installed if you finished Harvest )

Command

curl -O http://soracom-files.s3.amazonaws.com/handson/ultrasonic_funnel.py
python ultrasonic_funnel.py

Result

pi@raspberrypi:~ $ curl -O http://soracom-files.s3.amazonaws.com/handson/ultrasonic_funnel.py
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1029  100  1029    0     0    572      0  0:00:01  0:00:01 --:--:--   572
pi@raspberrypi:~ $ python ultrasonic_funnel.py
- reading distance
distance: 155.2 cm
- sending data to funnel
<Response [204]>

- reading distance
distance: 154.9 cm
- sending data to funnel
<Response [204]>

- reading distance
distance: 151.2 cm
- sending data to funnel
<Response [204]>

The script is sending data every 5 seconds (by default, can be changed by the 1st argument). Let's check it in AWS IoT Test tool.

Troubleshoot

If you see 400 error, you might need to check...

  • Is the SIM associated with the configuration group?
  • Is the Funnel configuration enabled?

Check data sent to AWS IoT

Open AWS IoT console and click on "Test" in the left menu. You will see MQTT client tool like below. Enter '#' as Subscription topic ('#' is wildcard for MQTT topic) and click "Subscribet to topic".

AWS Iot: Subscribe

If everything went fine, you will see published messages.

AWS Iot: Incoming data

You can see the incoming message includes original data sent from the device, and additional information like SIM card unique ID (IMSI). You can use it to tell which data came from which device.

Troubleshoot

If you do not see incoming data, check following.

  • Does the IAM user you created have "AWSIoTDataAccess" permission?
  • Is the region of the console same as the region your device is sending data to?

Create "Rule"

With AWS IoT, you can create "Rule" to pass data to other AWS services, such as...

  • Store data to Amazon S3 bucket
  • Save data to DynamoDB table
  • Invoke Lambda function etc...

In this section, create a rule to send data to CloudWatch as custom metrics.

Open AWS IoT Console and click "Act" in left menu, then click "Create a rule" or "Create" button.

You will see window like below. Input following fields.

  • Name: CloudWatch (or whatever you like)
  • Attribute: *
  • Topic filter sensor

AWS Iot: Rule

Click "add action" and choose "Send message data to CloudWatch" and click "Configure action".

Input following fields.

  • Metric name: distance
  • Metric namespace: sensor
  • Unit: None
  • Value: ${payloads.distance}

AWS Iot: Configure action

Then, click "Create a new role" and input name for newly created role, then select it in "IAM role name" list and click "Add action", finally click "Create rule".

Check data in CloudWatch Console

Open CloudWatch management console and click "Metrics" in the left menu. You will see "sensor" in the "Custom Namespaces".

If you don't see it, it may take a while until it shows up. Please wait for a several minutes in that case.

CloudWatch: metrics

Click on "sensor" -> "Metrics with no dimensions", then check "distance". You will see a graph in above.

CloudWatch: graph

You can change tab to "Graphed metrics" to select "Period" or "Statistic" to change view.

More things to Try

If you have time, you could try other AWS servies.

  • create cloudwatch alarm to send notification to your email address.
  • create other rules to store data to S3 and visualize on QuickSight or Athena

Cleanup

"Funnel" itself is not charged if you don't send any data to the entry point. However, you can disable or remove setting in group configuration to avoid unexpected charge.