Implementing time-critical cloud-to-device IoT message patterns on AWS IoT Core

Introduction

Widely adopted Internet of Things (IoT) communication standards for device-to-cloud and cloud-to-device are typically asynchronous, enabling event-driven patterns to ensure resilience, cost savings, and failure tolerance. However, customers across industries need to enable synchronous communication patterns to ensure time-critical logic in their edge devices.

Automotive manufacturers, for example, want their connected vehicles to be operated remotely and ensure their customers can trigger operations such as lowering side-windows or deactivating a car alarm in a timely and efficient manner. This typically requires that messaging between the vehicle and the cloud happens in real time in order to execute the customer’s command. However, this process is typically challenging to achieve with classic asynchronous IoT communication patterns, which are designed for disconnected scenarios where messages can persist until the device comes back online.
In the case of synchronous communications, this persistence is not required, and instead, messages must be executed immediately, otherwise they are discarded.

This post will walk you through the implementation of a synchronous communication pattern on AWS using AWS IoT capabilities. Customers will use an HTTP client to call an Amazon API Gateway endpoint. This action forwards the request to an AWS Lambda function, that will invoke logic running into edge applications.

Solution overview

We are proposing a solution where there is an application running on an IoT device which performs some tasks and returns a response after the execution ends. This solution enables an HTTP client to perform a request to the IoT device, waiting for immediate feedback. The application also needs to execute within a specific time window before a set time-out time, otherwise, it will fail, returning an HTTP error to the client.

The following steps represent the flow of a generic request that starts from an HTTP client and returns when when the device logic consumes it, or fails if no response is returned after a specific timeout (e.g. the device is not connected or there is no edge logic implemented for that specific request).

The HTTP client performs a request to your Amazon API Gateway instance, which exposes your AWS Lambda function externally.
The Amazon API Gateway instance forwards the request to your AWS Lambda function.
The AWS Lambda function creates an MQTT client instance, which will be used as the channel to exchange the HTTP request received with the AWS IoT Core instance.
Once the MQTT client is connected to your AWS IoT Core instance, it will forward the request to the AWS IoT Core topic, which is dedicated to exchanging the request’s payload with the device.
When the AWS IoT Core instance receives a request from the AWS Lambda function, that’s the moment where the synchronous approach is simulated according to the following steps:
1. AWS IoT Core forwards the MQTT request to the device.
2. The AWS Lambda function instance waits for a response from the device, or fails to a timeout.
3. The device reads the request, runs the business logic associated with it and creates the response.
4. The device publishes the MQTT payload containing the acknowledgment message and the optional response back to AWS IoT Core, to be forwarded back to the HTTP client.
The MQTT client on the AWS Lambda function receives the response from AWS IoT Core, containing the MQTT response.
The AWS Lambda function takes information from the MQTT response and generates the HTTP response to be returned to the Amazon API Gateway.
The Amazon API Gateway forwards the response to the client.
The client receives the HTTP response containing either the response generated by the device or the timeout generated by the AWS Lambda Function.

The following image shows the minimal architecture needed to implement this solution:

Implementing the solution in your AWS Account

Prerequisites

To execute this solution, you must satisfy the following prerequisites.

Knowledge of IoT Communication scenarios, MQTT protocol and event-driven/asynchronous patterns.
An AWS account.
A Linux-like terminal.
AWS Command Line Interface (CLI) installed. See AWS CLI Documentation for instructions on how to install and configure AWS CLI.
AWS SAM CLI. See AWS SAM CLI Documentation for instructions on how to install and configure AWS SAM CLI.
An AWS Identity and Access Management (IAM) user with the credentials for creating AWS resources through CLI.
A Python 3.9 environment installed on your machine.
A target IoT Device or a device simulator to implement the callback logic.
An HTTP client to execute tests. In this walk-through we shall use cURL.

Implementation specifications

In this walk-through we implement an example API for synchronous invocation that allows us to specify the following:

Method: Indicates an action to execute on the target device.
Target: Indicates the target device where we want to execute our action (eg. the ThingName).
Timeout: Indicates the timeout that we want to have for the execution of the command and the response.

Topic structure:

Outbound topic: sends a request to the IoT device.
ACK Topic: sends the ack from the IoT device to the AWS Lambda function.

The levels that compose the topics are the following:

target: the target device (eg. ThingName), inherited from the Target parameter from API call. The target device must subscribe to messages in the root topic space “{target}/#”
method: the identifier of the method to execute, inherited from the Target parameter from the API call. This identifier must be recognized by the device, in order to provide a response.
client_prefix: an identifier for the api in the cloud (eg. “invocation“), constantly defined within the application stack as Lambda environment variable
m_id: a random generated value for each new request through a UUID4, which ensures that multiple requests sent to the same device are managed separately by each requester.

The setup for implementing this solution involves the following steps:

Step 1: Deployment of an AWS Lambda function implementing a synchronous client to AWS IoT Core.
Step 2: Deployment of an HTTP API through Amazon API Gateway and integration with a Lambda function as the back-end.
Step 3: Get the AWS Lambda function endpoint.

As part of this blog post, we propose a deploy-able sample implementation for Steps 1 and 2 to use for testing purposes in a non-production account.

Implementation design principles

The following section summarizes the main design principles of the Lambda back-end of our proposed API:

AWS IoT Device SDK 2 for Python is used for a simple implementation of connecting, subscribing, and publishing of MQTT messages through Websockets.
A Websockets based MQTT client is used due to the simpler and more efficient authentication mechanism provided through IAM instead of X.509 certificates. This approach is advantageous as you don’t have to maintain the lifecycle of a client certificate for connecting to AWS IoT Core.

You can explore the IAM Role associated with the AWS Lambda function by visiting the template.yml file in the GitHub repository.

The application is deployed using AWS Serverless Application Model (SAM), which simplifies the process of defining cloud-native applications based on AWS Lambda and Amazon API Gateway.

Deploying an AWS IoT Core synchronous client in AWS Lambda

The following steps will guide you on creating an AWS Lambda function that is able to communicate with your AWS IoT Core instance through the MQTT protocol with Websockets.

Step 1 – Setup environmental variables

1 – Open the command line terminal window.

2 – Create a working directory in your machine that will be used to run commands. We will take /tmp/aws as an example for the rest of the blog post.

3 – Checkout the following repository into your working directory /tmp/aws:

git clone https://github.com/aws-samples/time-critical-cloud-to-device-iot-message-patterns-on-aws-iot-core iot_direct_invocations

The result should be that you have blog post contents into /tmp/aws/iot_direct_invocations.

4 – Define AccountID and Region environmental variables which will be used in the next steps.

AccountID: run the following command to get the AWS Account ID configured in your awscli:

aws sts get-caller-identity

You should receive a response like the following (in case of errors, please refer to the “Configure the AWS CLI” guide):

{
    "UserId": "AIDASAMPLEUSERID",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/DevAdmin"
}

Copy the “Account” field value and export it into the environmental variable:

export AccountID="123456789012"

Region: choose the region where you want to work (e.g. eu-west-1) and use the following command to export it:

export Region="eu-west-1"

5 – Define FunctionName environmental variable that will specify the name of your AWS Lambda function instance (e.g. iot-lambda-function):

export FunctionName="iot-lambda-function"

Note: be sure that you don’t already have an AWS Lambda function with this name or you will get an error in the deployment phase due to name collision.

6 – Define IoTCoreEndpoint environmental variable that contains the URL of the AWS IoT Core instance that you will use later on.

Run the following command:

aws iot describe-endpoint --endpoint-type iot:Data-ATS

You should get a response like the following:

{
    "endpointAddress": "<instance_id>-ats.iot.<your_region>.amazonaws.com"
}

Copy the endpointAddress value and export it like this:

export IoTCoreEndpoint="<instance_id>-ats.iot.<your_region>.amazonaws.com"

Step 2 – Deploy the AWS Lambda function

Now that your environment is configured to deploy the AWS Lambda function in your AWS Account, open the terminal and place the prompt into the directory /tmp/aws/iot_direct_invocations/sam/iot-lambda-client/˛

Inside this folder, you will find the SAM artifacts and templates which will be used to deploy the following elements:

IAM roles and policies.
API Gateway instance.
AWS Lambda function with code (you will find the Python code in the /tmp/aws/iot_direct_invocations/sam/iot-lambda-client/)˛

1 – Build the CloudFormation template with SAM: before deploying all the elements needed by the AWS Lambda function to be exposed, you need to build the resulting CloudFormation template with the following command:

sam build

If you see the statement Build Succeeded, you are ready to go to the next stage.

2 – Deploy the CloudFormation stack: once the build process completes successfully, you are ready to deploy the AWS Lambda function environment with the following command:

sam deploy \
    --resolve-s3 \
    --stack-name iot-lambda-client \
    --capabilities CAPABILITY_IAM \
    --parameter-overrides \
        AccountID=${AccountID} \
        Region=${Region} \
        LambdaName=${FunctionName} \
        IoTCoreEndpoint=${IoTCoreEndpoint} \
        ClientIDPrefix=invocation_client

The output of the command will tell you if the process ended successfully.

Note: the command is idempotent, so you can run it as many times as you want and the result will not change as long as the SAM templates/artifacts don’t change. However, in this case, the output will report a message like Error: No changes to deploy. Stack iot-lambda-client is up to date, this does not mean that your deployment failed, it means that there were no changes to deploy.

Step 3 – Get the AWS Lambda function endpoint

After deploying your AWS Lambda function, you need to retrieve the URL to use with you HTTP client in order to invoke it. To do that, use the following command:

aws cloudformation describe-stacks --stack-name iot-lambda-client

This command retrieves information that describes the AWS CloudFormation stack with name iot-lambda-client, which is the one created by SAM in the previous step. The output represents a json object containing the nested array “.Stacks[].Outputs”. Inside this array, you will find some key-value pairs, look for "OutputKey": "InvokeApi" and get the value of the related "OutputValue". You should find something like this:

"https://<id>.execute-api.<region>.amazonaws.com/Prod/invoke/"

This is the URL you will use to invoke the AWS Lambda function from internet. Export it in an environment variable in order to be used later on tests:

export APIEndpoint="https://<id>.execute-api.<region>.amazonaws.com/Prod/invoke/"

Testing the solution

Testing the solution requires two different steps:

Creating an instance of a simulated AWS IoT Thing device listening for a direct request
Performing an authenticated request to the AWS Lambda function.

Preparing the Simulated AWS IoT Thing device

An effective way to test the solution is to create a software client which will simulate interactions with our scenario, connecting to the AWS IoT Core instance described above. This client will be represented by an AWS IoT thing and will respond to invocations coming from the AWS Lambda function.

In our example, we set up an IoT thing in the AWS IoT Core registry and associate a device certificate and an IoT policy to the IoT thing. The device certificate and the device private key will be provided to the device to communicate with AWS.

As a best practice, a real production provisioning flow should allow you to avoid sharing the private key over the public internet, and it is advised that you embed a provisioning flow as part of your IoT device design.

AWS has a list of options for device provisioning as part of AWS IoT Core documentation and a whitepaper on “Device Manufacturing and Provisioning with X.509 Certificates in AWS IoT Core” that explains in depth each option in respect to real customer scenarios.

1 – Go the the device simulator’s working directory /tmp/aws/iot_direct_invocations/sam/test-client/.

2 – Open a command line terminal window and run the following command to generate a device certificate and a key pair, the files will be created in the working directory. Copy your certificateArn and certificateId from the output of the command:

aws iot create-keys-and-certificate \
    --certificate-pem-outfile "TestThing.cert.pem" \
    --public-key-outfile "TestThing.public.key" \
    --private-key-outfile "TestThing.private.key" \
    --region ${Region} \
    --set-as-active

The output of the previous command should be something like the following:

{
    "certificateArn": "arn:aws:iot:<region>:<account_id>:cert/<certificate_id>",
    "certificateId": "<certificate_id>",
    "certificatePem": "-----BEGIN CERTIFICATE-----\n<certificate_data>\n-----END CERTIFICATE-----\n",
    "keyPair": {
        "PublicKey": "-----BEGIN PUBLIC KEY-----\n<public_key_data>\n-----END PUBLIC KEY-----\n",
        "PrivateKey": "-----BEGIN RSA PRIVATE KEY-----\n<private_key_data>\n-----END RSA PRIVATE KEY-----\n"
    }
}

Take note of the certificateArn and export it to an environmental variable in order to use it in the next step:

export CertificateArn="arn:aws:iot:<region>:<account_id>:cert/<certificate_id>"

Note: according to AWS security policies, the private key will never be transferred to the cloud environment. If you lose it, you’ll need to generate it again.

3 – Now it’s time to prepare the SAM script, which will create what you need to prepare the AWS IoT thing representing the simulator device. To do that, you need to open the terminal windows and go to the working directory /tmp/aws/iot_direct_invocations/sam/test-client/ and build the SAM template:

sam build

4 – Once the build process terminates without issues, you can proceed with the deployment of the AWS CloudFormation template generated in the previous step. To do that, deploy the IoT thing creation with SAM using this command:

sam deploy \
     --resolve-s3 \
    --stack-name test-iot-thing \
    --parameter-overrides \
        Region=${Region} \
        AccountID=${AccountID} \
        ClientIDPrefix=invocation_client \
        TestingIoTThingCertificateARN=${CertificateArn}

You should see the registered thing, policy and device certificate in your AWS account through the AWS IoT Console.

5 – Once the AWS IoT thing device is provisioned into the cloud environment, you just need to run the simulator. To do that, run the following command:

python index.py \
    --endpoint ${IoTCoreEndpoint} \
    --cert TestThing.cert.pem \
    --key TestThing.private.key \
    --client-id Device001

If everything has been set up correctly, you should see the client stopping to the message Waiting for command .... This python script is simulating a device waiting for a command to be forwarded to the AWS IoT Core instance.

Perform an authenticated request to the AWS Lambda function endpoint

You need to perform an HTTP request towards the Amazon API Gateway endpoint which is exposing the AWS Lambda function API to test the interaction with the simulated device defined in the previous step.

1 – Prepare the Amazon API Gateway endpoint URL:

The URL representing the GET request to be performed will be composed like this:

export ENDPOINT="${APIEndpoint}?request=my_request_1&method=reqid-ABCD&target=Device001&timeout=3"

Note the following parameters:

request: it’s a string representing the value of the request passed to the AWS Lambda function that is forwarded directly to the device’s logic. The AWS Lambda function does not enter into the merits of the parameter value. We suggest to pass a base64 encoding in order to avoid issues with URL compatible parameters in the request.
method: it represents the identifier of the method triggered into the device’s logic.
target: it’s the client id used to map the request to a specific device. It must be the same used on step 5 (–client-id).
timeout: it’s the value of the period of time the AWS Lambda function will wait the device to respond before returning a timeout to the caller.

2 – Prepare the authenticated request:

As a security best practice, you should never expose APIs without any kind of authentication. That’s why in this example, we deployed the API endpoint using the AWS IAM authentication mechanism. This basically means that the exposed invoke API resource is configured to accept execute-api:Invoke requests only from IAM users which have proper policy attached to them.

What you need to do ensuring that the IAM credentials that you are using in this example are associated to an IAM user which has proper policies attached as per this documentation. Then, every request performed to the endpoint generated in step 1 must be properly signed according to the Signature Version 4 signing process.

3 – Use the http python client to perform the request:

For you convenience, we provided a tool which implements the logic behind the signing process. It’s a Python script called perform_authenticated_request.py which can be found in the repository root folder /tmp/aws/iot_direct_invocations/.

So, what you need to do now is placing the terminal in the root folder of the repository (e.g. /tmp/aws/iot_direct_invocations/) and run the following command to perform an authenticated request:

python perform_authenticated_request.py $ENDPOINT

The output of this request depends on the following scenarios:

The device simulator is connected:

{
    "result": "ok",
    "elapsed": "383",
    "response": "Your request was 'my_request_1'. Some random response: '333.0155808661127'"
 }

The device simulator is not connected:

{
    "result": "timeout",
    "elapsed": "3795"
}

Cleaning up your resources

To help prevent unwanted charges to your AWS account, you can delete the AWS resources that you used for this walk-through. These AWS resources include the AWS IoT Core things and certificates, AWS Lambda function, and Amazon API Gateway. You can use the AWS CLI, AWS Management Console, or the AWS APIs to perform the cleanup. In this section, we will use the AWS CLI approach. If you want to keep these resources, you can ignore this section.

Cleaning up AWS IoT Core resources

1 – Detach the device certificate from the IoT policy:

aws iot detach-policy --target "arn:aws:iot:<your_region>:<your_account_ID>:cert/<certificate_id>" --policy-name "TestPolicy"

2 – Delete the IoT policy

aws iot delete-policy --policy-name TestPolicy

3 – Detach the device certificate from the test IoT thing

aws iot detach-thing-principal --thing-name TestThing --principal arn:aws:iot:<your_region>:<your_account_ID>:cert/<certificate_id>

4 – Delete the device certificate from AWS IoT Core

aws iot delete-certificate --certificate-id <certificate_id>

5 – Delete the IoT thing from AWS IoT Core

aws iot delete-thing --thing-name TestThing

Cleaning up API Gateway and Lambda resources via SAM

Delete the SAM stack associated with the synchronous invocation resources

sam delete --stack-name iot-lambda-client

Conclusion

In this post, we discussed some of the main challenges customers face when a synchronous pattern is required for time-critical communication scenarios. The architecture and implementation proposed in this blog shows a test artifact that you can adopt as a baseline to implement such a feature in your solution.

To learn more about how to use AWS IoT Core, you can refer to the documentation.

AWS welcomes feedback. Please connect with us on LinkedIn if you have thoughts or questions.

The Internet of Things on AWS – Official Blog