Using Bluetooth Low Energy with Amazon FreeRTOS on Espressif ESP32

Today Amazon Web Services (AWS) announced the beta of Amazon FreeRTOS BLE, a feature that makes it possible for embedded developers to securely connect Amazon FreeRTOS devices that use Bluetooth Low Energy (BLE) to AWS IoT through Android or iOS devices. BLE support in Amazon FreeRTOS lets developers create new applications for devices that need lower power than any other forms of connectivity, including Wi-Fi.

With BLE support in Amazon FreeRTOS, developers can use the standard Generic Access Profile (GAP) and Generic Attributes (GATT) profiles through a universal API layer to create BLE applications that are portable across any Amazon FreeRTOS-qualified device and use companion Android and iOS SDKs to integrate with AWS IoT functionality. According to the BLE specifications, GAP defines how BLE devices broadcast availability and communicate with each other. GATT describes how data is transferred once a connection is established.

Getting Started with Amazon FreeRTOS BLE

In this post, I provide a use case of a BLE device connecting to AWS IoT through an Android proxy. This allows the BLE device to use the same MQTT protocol, agnostic to the underlying communication carrier of either BLE or Wi-Fi. Because BLE offers lower power compared to Wi-Fi, devices can use the MQTT protocol to connect to AWS IoT services over BLE. This brings the best of low power and rich AWS IoT services, such as Amazon FreeRTOS over-the-air updates, to the devices in the field.

Connecting Espressif ESP32 to AWS IoT via BLE with Amazon FreeRTOS BLE

At the start of the cycle after reset, ESP32 is in advertising mode and uses the GAP layer to broadcast data out to nearby BLE devices. One of the nearby BLE devices is an Android app which, when started, is in scanning mode. Advertising mode is a one-to-many transfer and offers no guarantees about data coherence.

In the Android app it will scan for devices that match the name or address of the ESP32, and once the connect switch is toggled, both ESP32 and the Android app will enter into connected mode. BLE device will expose a custom GATT profile to allow the client to behave as a proxy and access AWS IoT services from the AWS cloud. During the authentication process the BLE device uses the AWS Cloud service certificate to securely wrap a challenge directed to the cloud service through the un-authenticated client. The challenge contains information about the BLE device and the client to be authenticated. If the challenge is satisfied, then the BLE device may grant the authenticated client proxy the ability to access AWS IoT services on behalf of the device. Data packets are delivered to the BLE device through a custom GATT profile.

These are the high level steps to connect a ESP32 to AWS IoT using the BLE proxy running on an Android phone:

AWS configuration – Create AWS IoT thing and policy, create a Cognito user pool and configure Cognito identity
ESP32 – Download, configure and run the Amazon FreeRTOS BLE on the ESP32
BLE pass-through app – Download, configure and run the Amazon FreeRTOS example MQTT proxy app on an Android phone

Prerequisite

ESP32 development board
MicroUSB to USB A cable
AWS account (Free Tier is sufficient)
Sufficient disk space (~500Mb) for the Xtensa toolchain and Amazon FreeRTOS source code and examples. This post is written with the assumption that Xtensa toolchain, ESP-IDF, and Amazon FreeRTOS code is installed in the esp directory in the user’s home directory, ~/esp. You must add ~/esp/xtensa-esp32-elf/bin to your $PATH variable.
An Android phone with Android 4.3 or later.
Android Studio with Android 4.3 or later (API Level 18 or later) SDK installed.

You can find information about the setup of the ESP32 hardware and build instructions in the Connect Microcontroller-Based Devices to the Cloud with Amazon FreeRTOS and Espressif ESP32 blog post written by my colleague, Anton Shmagin.

AWS Configuration

AWS IoT configuration

First, I set up the AWS IoT thing and policy. Because I am using the MQTT proxy on my Android phone, which will be authenticated using Amazon Cognito, my ESP32 device does not need the AWS IoT certificates.

Step 1 – Create an AWS IoT Policy

Before you create the AWS IoT Policy, you need to know your AWS region and AWS account number.

Sign in to the https://console.thinkwithwp.com/iot/.
From the upper-right corner of the AWS Management Console, choose My Account. Under Account Settings, make a note of the 12-digit account ID.
Browse to the AWS IoT console.
Browse to the AWS IoT Core console.
From the left navigation pane, choose Settings. In Custom endpoint, make a note of the endpoint value.
The endpoint should be something like “xxxxxxxxxxxxxx.iot.us-west-2.amazonaws.com”, in this case, the AWS region will be us-west-2.
From the left navigation pane, choose Secure, choose Policies, and then choose Create.
If you do not have any policy created in your account, you will see the message “You don’t have any policies yet” is displayed. Choose Create a policy.
Enter a name for your policy (for example, esp32_mqtt_proxy_iot_policy).
In the Add statements section, choose Advanced mode. Copy and paste the following JSON into the policy editor window. Replace the aws-account-id with your account ID (step 2). Replace aws-region with us-west-2 (step 4).
Choose Create.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:Connect",
      "Resource": "arn:aws:iot:<aws-region>:<aws-account-id>:*"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Publish",
      "Resource": "arn:aws:iot:<aws-region>:<aws-account-id>:*"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Subscribe",
      "Resource": "arn:aws:iot:<aws-region>:<aws-account-id>:*"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Receive",
      "Resource": "arn:aws:iot:<aws-region>:<aws-account-id>:*"
    }
  ]
}

Step 2 – Create an AWS IoT Thing

To create an AWS IoT thing:

Sign in to the https://console.thinkwithwp.com/iot/.
From the left navigation pane, choose Manage, and then choose Thing. From the top-right corner, choose Create.
If you do not have any IoT things registered in your account, the message “You don’t have any things yet” is displayed. Choose Register a thing.
On the Creating AWS IoT things page, choose Create a single thing.
On the Add your device to the thing registry page, enter a name for your thing (for example, esp32-ble). Only alphanumeric, hyphen (“-“) and underscored (“_”) are allowed. Choose Next.
On the Add a certificate for your thing page, under Skip certificate and create thing, choose Create thing without certificate.

Because I am using the BLE proxy mobile app that uses an Amazon Cognito credential for authentication and authorization, no device certificate is required.

AWS Cognito Configuration

Amazon Cognito is required for authentication of the MQTT proxy mobile app. An IAM policy is attached to the authenticated identity to allow the principal to attach the AWS IoT policy to the credential.

Step 1 – AWS Cognito user pool

Sign in to the Amazon Cognito console at https://console.thinkwithwp.com/cognito/users/.
From the right top navigation banner, choose Create a user pool.
Enter the pool name (for example, esp32_mqtt_proxy_user_pool.
Choose Review defaults.
In the App Clients, click Add app client, and then choose Add an app client.
Enter a app client name (for example mqtt_app_client).
Make sure the Generate client secret is selected.
Choose Create app client.
Choose Return to pool details.
On the Review page of the user pool, choose Create pool.
You should see a message that says “Your user pool was created successfully.”
Make a note of the pool ID.
In the navigation pane, choose App clients.
Click Show Details.
Make a note of the app client ID and the app client secret.

Step 2 – Create An AMAZON Cognito Identity Pool

Sign in to the Amazon Cognito console at https://console.thinkwithwp.com/cognito/federated
Choose Create new identity pool.
Enter a name for the identity pool (for example, mqtt_proxy_identity_pool).
Expand Authentication providers.
Choose the Cognito tab.
Enter the user pool ID and app client ID from the previous step.
Choose Create Pool.
On the next page, choose Allow to create new roles for authenticated and unauthenticated identities.
Make note of the Identity pool ID, which is in the format of us-east-1:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Step 3 – Attach IAM policy to the authenticated identity

Attach an IAM policy to the authenticated identity so that the credential can attach the IoT policy to it.

Open the Amazon Cognito console at https://console.thinkwithwp.com/cognito/federated
Choose the identity pool that you just created (in my example, mqtt_proxy_identity_pool).
Choose Edit identity pool.
Make note of the IAM Role assigned to the Authenticated role (for example, Cognito_mqtt_proxy_identity_poolAuth_Role).
Open the IAM console at https://console.thinkwithwp.com/iam/home
In the navigation pane, choose Roles.
Search for the role (in my example,Cognito_mqtt_proxy_identity_poolAuth_Role), and then choose it.
Choose Add inline policy, and then choose JSON.

Enter the following policy:

{
   "Version": "2012-10-17",
   "Statement": [
   {
      "Effect": "Allow",
      "Action": [
         "iot:AttachPolicy",
         "iot:AttachPrincipalPolicy",
         "iot:Connect",
         "iot:Publish",
         "iot:Subscribe"
      ],
      "Resource": "*"
   }]
}

Choose Review Policy
Enter a policy name (for example, mqttProxyCognitoPolicy)
Choose Create policy

Setting up ESP32 with Amazon FreeRTOS BLE

Hardware Setup

For this post, I use the ESP32-DevKitC V4 developed by Espressif.

Make sure that drivers for the board are installed. If you are installing the driver for Silicon Labs CP2104 on macOS High Sierra, you might need to restart installation after you whitelist the installation package in System Preferences > Security & Privacy.
Connect the ESP32 development board
The UART to USB should appear as follows.
1. On Windows the driver would enumerate the port as COMx:
2. On macOS:
```
$ ls /dev/tty.S*
/dev/tty.SLAB_USBtoUART
```
3. On Linux:
```
$ ls /dev/ttyUSB* /dev/ttyUSB0
```

Download instructions for Amazon FreeRTOS BLE for ESP32

Clone the Amazon FreeRTOS from GitHub repository at https://github.com/aws/amazon-freertos . The branch is feature/ble-beta
```
$ cd ~/esp 
$ git clone -b feature/ble-beta --single-branch https://github.com/aws/amazon-freertos
```
Set your baud rate and serial port. The example Makefile for respective hardware is in amazon-freertos-library-share/demos/<manufacturer>/<hardware kit name>/make/.
For Espressif ESP32 it is in amazon-freertos-library-share/demos/espressif/esp32_devkitc_esp_wrover_kit/make/
```
$ cd ~/esp/amazon-freertos-library-share/demos/espressif/esp32_devkitc_esp_wrover_kit/make
$ make menuconfig
```
Save the configuration

Configure Amazon FreeRTOS

Open amazon-freertos-library-share/demos/espressif/esp32_devkitc_esp_wrover_kit/common/application_code/main.c. Make note of the mainBLE_DEVICE_NAME. You will need it for the Android app to match the BLE name (for example, ESP32).
Open ~esp/amazon-freertos-library-share/demos/common/include/aws_clientcredential.h
Change the following:
1. clientcredentialMQTT_BROKER_ENDPOINT to the AWS IoT Endpoint
2. clientcredentialIOT_THING_NAME to the AWS IoT Thing you created earlier (in my example, esp32-ble).

Compile and Flash the Amazon FreeRTOS onto the ESP32

Flash the program to ESP32
```
$ make flash monitor
```

If the program is successfully flashed, you should see this in your terminal

I (719) wifi: Init dynamic tx buffer num: 32
I (719) wifi: Init data frame dynamic rx buffer num: 32
I (719) wifi: Init management frame dynamic rx buffer num: 32
I (729) wifi: Init static rx buffer size: 1600
I (729) wifi: Init static rx buffer num: 10
I (739) wifi: Init dynamic rx buffer num: 32
0 49 [Btc_task] Started advertisement. Listening for a BLE Connection.

Download, configure and run the example Android app for MQTT Proxy

Clone the example Amazon FreeRTOS MQTT proxy app for Android from the GitHub repository at https://github.com/aws/amazon-freertos-ble-android-sdk
```
$ cd ~/android
$ git clone https://github.com/aws/amazon-freertos-ble-android-sdk
```
Choose Open an existing Android Studio project to open the project from Android Studio
Navigate to and open ~/android/amazon-freertos-ble-android-sdk
Open amazonfreertossdk/src/main/java/com/amazon/aws/amazonfreertossdk/AmazonFreeRTOSManager.java and edit TEMPORARY_IOT_ENDPOINT to use the AWS IoT endpoint
Open app/src/main/java/com/amazon/aws/freertosandroid/AuthenticatorActivity.java and edit the following:
1. For AWS_IOT_POLICY_NAME, use the name of the IoT Policy (in my example, esp32_mqtt_proxy_iot_policy).
2. ForCOGNITO_POOL_ID, use the pool ID
3. ForAWS_REGION, use the region of the AWS services
Open app/src/main/java/com/amazon/aws/freertosandroid/MainActivity.java and edit ESP_NAME.
In our example, the ESP_NAME is ESP32

Open app/src/main/res/raw/awsconfiguration.json, and edit the file

{
  "UserAgent": "MobileHub/1.0",
  "Version": "1.0",
  "CredentialsProvider": {
    "CognitoIdentity": {
      "Default": {
        "PoolId": "<Cognito Identity Pool ID, e.g. us-east-2:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx>",
        "Region": "<AWS region, e.g. us-east-2>"
      }
    }
  },
  "IdentityManager": {
    "Default": {}
  },
  "CognitoUserPool": {
    "Default": {
      "PoolId": "<Cognito user pool ID, such as us-east-2_0000XXXXX>",
      "AppClientId": "<Cognito user pool app client ID>",
      "AppClientSecret": "<Cognito user pool app client secret>",
      "Region": "<AWS region, e.g. us-east-2>"
    }
  }
}

Connect your Android phone to your computer.
Goto Run → Run ‘app’.
In Select Deployment Target, choose the phone in Connected Devices and choose OK.

MQTT Proxy

Registration and login

If the Android app compiles and launches successfully on your Android phone, you should see the following sign-in screen:
Choose Create New Account. Enter your email address and password to create an Amazon Cognito user.
Amazon Cognito will send a confirmation code to your email address.
On the next screen, enter the confirmation code sent to your email address.

Run the MQTT proxy

Choose Scan to scan for nearby BLE devices
After a device that matches the ESP_ADDR or ESP_NAME is found, the Connect switch is enabled.
Toggle the Connect switch

After the Android app is connected to the ESP32, you should be able to see these messages in the ESP32 terminal. You can ignore the MQTT error for now, because you activate the MQTT proxy next.

2 16681 [Btc_task] BLE Connected to remote device, connId = 0
3 16682 [Tmr Svc] Creating MQTT Echo Task...
4 16682 [MQTTEcho] MQTT echo attempting to connect to a1jgguqwhae8ke-ats.iot.us-east-2.amazonaws.com.
5 16682 [MQTTEcho] Sending command to MQTT task.
6 16683 [MQTT] Received message 10000 from queue.
7 16683 [MQTT] Failed to send notification, mqtt proxy state:1 
8 16683 [MQTT] Failed to send CONNECT message, sent = 0
9 16683 [MQTT] MQTT_Connect failed!

Choose Discover to discover and register all GATT services from the esp32.
Choose Set MTU to increase the default MTU from 20 to 500.
Toggle the MQTT Proxy switch to enable the MQTT proxy.

After the MQTT proxy is enabled, you should see these messages in the ESP32 terminal:

65 4316 [MQTTEcho] MQTT echo attempting to connect to a1jgguqwhae8ke-ats.iot.us-east-2.amazonaws.com.
66 4316 [MQTTEcho] Sending command to MQTT task.
67 4316 [MQTT] Received message 70000 from queue.
68 4446 [Btc_task] MQTT Connect was accepted. Connection established.
69 4446 [Btc_task] Notifying task.
70 4446 [MQTTEcho] Command sent to MQTT task passed.
71 4446 [MQTTEcho] MQTT echo connected.
72 4446 [MQTTEcho] Sending command to MQTT task.
73 4446 [MQTT] Received message 80000 from queue.
74 4558 [Btc_task] MQTT Subscribe was accepted. Subscribed.
75 4558 [Btc_task] Notifying task.
76 4558 [MQTTEcho] Command sent to MQTT task passed.
77 4558 [MQTTEcho] MQTT Echo demo subscribed to freertos/demos/echo
78 4558 [MQTTEcho] Sending command to MQTT task.
79 4559 [MQTT] Received message 90000 from queue.
80 4582 [Btc_task] MQTT Publish was successful.
81 4582 [Btc_task] Notifying task.
82 4583 [MQTTEcho] Command sent to MQTT task passed.
83 4583 [MQTTEcho] Echo successfully published 'Hello World 0'
84 4593 [Echoing] Sending command to MQTT task.
85 4593 [MQTT] Received message a0000 from queue.
86 4617 [Btc_task] MQTT Publish was successful.
87 4617 [Btc_task] Notifying task.
88 4617 [Echoing] Command sent to MQTT task passed.
89 4619 [Echoing] Message returned with ACK: 'Hello World 0 ACK'

Android Studio output

If you have Android Studio connected to your phone, you can filter the messages in the Logcat by choosing Show only selected application.

2018-11-09 11 (tel:2018110911):50:31.398 26628-26643/com.amazon.aws.freertosandroid D/AmazonFreeRTOSManager: Characteristic changed for: MQTT_TX with data: {"type":3,"topic":"ZnJlZXJ0b3MvZGVtb3MvZWNobw==","qoS":1,"msgID":272,"payloadVal":"SGVsbG8gV29ybGQgNDI="}
2018-11-09 11 (tel:2018110911):50:31.405 26628-26643/com.amazon.aws.freertosandroid I/AmazonFreeRTOSManager: Received Mqtt Message type : 3
2018-11-09 11 (tel:2018110911):50:31.419 26628-26643/com.amazon.aws.freertosandroid I/AmazonFreeRTOSManager: Sending mqtt message to IoT on topic: freertos/demos/echo message: Hello World 42
2018-11-09 11 (tel:2018110911):50:31.524 26628-29470/com.amazon.aws.freertosandroid I/AWSIotMqttManager: delivery is complete
2018-11-09 11 (tel:2018110911):50:31.524 26628-29470/com.amazon.aws.freertosandroid D/AmazonFreeRTOSManager: Publish msg delivery status: Success
2018-11-09 11 (tel:2018110911):50:31.524 26628-29470/com.amazon.aws.freertosandroid I/AmazonFreeRTOSManager: Sending PUB ACK back to device.
2018-11-09 11 (tel:2018110911):50:31.532 26628-29470/com.amazon.aws.freertosandroid D/AmazonFreeRTOSManager: Processing BLE command: WRITE_CHARACTERISTIC queue size: 0
2018-11-09 11 (tel:2018110911):50:31.534 26628-29470/com.amazon.aws.freertosandroid D/AmazonFreeRTOSManager: Writing to characteristic: MQTT_RX with data: {"msgID":272,"type":4}
2018-11-09 11 (tel:2018110911):50:31.634 26628-26643/com.amazon.aws.freertosandroid D/AmazonFreeRTOSManager: onCharacteristicWrite for: MQTT_RX; status: Success
2018-11-09 11 (tel:2018110911):50:31.634 26628-26643/com.amazon.aws.freertosandroid D/AmazonFreeRTOSManager: There's no ble command in the queue.

Monitor the MQTT messages in AWS IoT MQTT client

You can monitor the MQTT messages are being received in AWS IoT using the MQTT test client

Sign in to the AWS IoT console at https://console.thinkwithwp.com/iot
In the navigation pane, choose Test.
Subscribe to the freertos/demos/echo.
You should be able to see the MQTT messages in the AWS IoT console.

Conclusion

Getting started with Amazon FreeRTOS BLE is easy. You can download source code from the Amazon FreeRTOS console and the SDK for either Android or iOS from GitHub. The download also contains a sample Android/iOS app that you can use to start building your BLE device. When the BLE device boots up, it enters the standard BLE pairing process for headless devices.

Amazon FreeRTOS support for BLE is available for any of the qualified development boards, such as Espressif ESP32 used in this post. For more information about how to get started, see the Amazon FreeRTOS Developer Guide.

The Internet of Things on AWS – Official Blog