The Internet of Things on AWS – Official Blog

Device Simulation with AWS IoT and AWS Lambda

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

The AWS IoT platform enables you to connect your devices and build scalable Internet of Things solutions with the breadth of Amazon Web Services.

When shopping for a new tool, many developers want to be able to test-drive options before making a choice. When evaluating an IoT solution, it’s not practical to do so at scale with physical devices. Building a sensor simulator is the next best choice. With a sensor simulator built on top of AWS Lambda, you can elastically generate device sensors that report their state to the cloud.

Lambda Simulator Architecture

In this blog post, you will learn how to implement a simulator that feeds data sets into AWS IoT. You’ll see how to monitor the simulators in real time, visualize data, configure your own sensor, and add hooks to start simulators from your projects.

Why Simulate? Why Lambda?

One device is cheap, but you’re building at a bigger scale than that. What will data for all of your devices look like in AWS? Simulation is an inexpensive means to test a cloud platform at scale, without first making changes to your real devices.

AWS Lambda is a perfect partner for simulation. On-demand, serverless compute time lets you try out a fleet of devices on AWS IoT with ease. In the Lambda function provided in this post, you’ll be able to specify the following each time you run the function:

  • number of devices to simulate
  • simulation duration
  • publish frequency
  • message topic
  • data set
  • Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) domain

The simulator connects to AWS IoT and publishes data as messages over MQTT. You can simulate many devices with the same configuration, or invoke multiple Lambda functions with unique configurations. You can test the functioning of AWS IoT policies by making changes to your Lambda execution role.

Running the AWS CloudFormation Template

As part of this blog post, we are providing an AWS CloudFormation template to get you up and running with Lambda simulators quickly. This template creates a stack of AWS resources that do the following:

  • a Lambda function that simulates devices publishing to AWS IoT.
  • an Amazon ES domain for visualizing published data.
  • an AWS IoT rule that forwards simulator messages to Amazon ES.
  • an Amazon S3 bucket to store your simulator data sets.
  • all required AWS Identity and Access Management (IAM) roles.

Note: This CloudFormation template is written to be executed in the US West (Oregon) Region. When creating Lambda functions through CloudFormation, the Lambda simulator source must be hosted in the same region as the S3 bucket. To use this template in another AWS region, download the Lambda simulator source, and you can update the CloudFormation template to point to a bucket you own with the source there.

To run the CloudFormation template and create these resources:

  1. Sign in to the AWS Management Console and open the AWS CloudFormation console at https://console.thinkwithwp.com/cloudformation/home?region=us-west-2.
  2. Choose Create Stack.
  3. In the Template section, select Specify an Amazon S3 template URL to type ot paste the following URL, and then choose Next.
    https://s3-us-west-2.amazonaws.com/iot-us-west-2-simulator-publicbucket/createSimulator.json
  4.  In the Specify Details section, type a name for your stack (for example, my-iot-simulator-stack).
  5. In the Parameters section, type a name for your S3 bucket (for example, YOURNAME-iot-simulator-data-sets), and then choose Next.
  6. On the Options page, choose Next.
  7. On the Review page, under Capabilities, select the I acknowledge that this template might cause AWS CloudFormation to create IAM resources check box, and choose Create.

CloudFormation will now create your AWS resources. It can take up to ten minutes to create an Amazon Elasticsearch Service domain.

Starting the Simulator

Until you’re familiar with the configuration parameters, you should start the simulator manually. To do so, navigate to the AWS Lambda console. The simulator will have the prefix of your CloudFormation stack (for example, my-iot-simulator-stack-AWSLambda-ABC123). In your function’s view, from the Actions drop-down menu, choose Configure test event.

This is where you will set the runtime configuration of your simulator. The following sample JSON configuration is followed by an explanation of options.

 {
  "esEndpoint": "search-my-iot-elastic-1ubx76h2qqoan-3dijocbhik332asd4sryz4au5q.us-west-2.es.amazonaws.com",
  "esDomainArn": "arn:aws:es:us-west-2:111122223333:domain/my-iot-elastic-1ubxg6r2qwean",
  "topic": "test/topic",
  "simTime": 10000,
  "interval": 500,
  "numDevice": 3
 }
  • esEndpoint: your Amazon ES domain endpoint.
  • esDomainArn: your Amazon ES domain.
  • topic: the AWS IoT topic this simulator will publish to.
  • simTime: the duration of this simulator, in milliseconds (max 240000).
  • interval: the interval between publishes, in milliseconds.
  • numDevice: the number of devices of this configuration to run.
  • bucketName: (OPTIONAL) the S3 bucket with your simulator data set to publish. Defaults to iot-us-west-2-simulator-publicbucket.
  • key: (OPTIONAL) the S3 key of your simulator data file to read for published messages. Defaults to physiological-data.csv.

Replace the values shown for esEndpoint and esDomainArn with those for your domain. You can find these values in the Amazon OpenSearch Service console after you click on your domain.

Note: If you omit the optional bucketName and key parameters, a data set file provided with the simulator will be used. It contains records of a sleep healthcare monitoring device. You can review this data set here.

In the AWS Lambda console, when you choose Save and test, your configuration JSON will be saved and the Lambda function will start with this configuration. If you scroll down, you’ll see Execution result: succeeded. Congratulations, you’ve run your first device simulator!

Visualize the Data

There are two ways to see the simulator data in AWS:

  • In real time, in the AWS IoT console.
  • In a data visualization dashboard called Kibana, which is included with the Amazon ES domain.

In the AWS IoT console, navigate to the MQTT Client. Enter or generate a client ID, and then choose Connect. Choose Subscribe to topic, enter the topic you specified in your Lambda configuration, and then choose Subscribe. If your Lambda simulator is running, you should now see these messages arriving at the interval you specified, times the number of devices specified. If your simulator has already run, in the AWS Lambda console, choose Test again to restart it.

To visualize the published data from the simulator in Kibana, open the Amazon OpenSearch Service console. Find your Amazon ES domain in the list and choose it. Open the Kibana link in a new tab.

On your first visit to Kibana, you’ll see a settings page where you can choose an index pattern. This tells Kibana which indices to surface in your queries. Clear the Index contains time-based events box (you may want this in the future for your own data sets), and in Index name or pattern, type “index_*”. Choose Create.

Choose the Discover tab to see your simulated data and start using Kibana. For more information about configuring visualizations of AWS IoT data in Kibana, see another of our blog posts.

Extending the Simulator

Now that you have a working device simulator, let’s break down the important bits of code in the simulator Lambda function in case you want to extend it. This simulator will read through the input sample data and publish one message per line, per device, until the end of file, duration, or maximum Lambda runtime is reached.

For example, if the input sample data is 10 lines long, the number of devices to simulate is 30, and the interval is 1000 ms, then over 10 seconds of runtime, this simulator will make 30 connections to AWS IoT and publish a total of 300 messages.

In index.js, the event handler calls createResources(Object, Function). This exists to ensure that your Amazon ES domain is ready and your AWS IoT rule is set up to feed it incoming published messages. It also fetches a CSV file from S3, which will be parsed as messages for your simulated devices to publish.

 createResources(event, (err, data) => {
   if (err) {
     console.log(err);
     return;
   }
   const s3 = new AWS.S3({
     region: 'us-west-2'
   });
   var bucket = event.bucketName;
   var key = decodeURIComponent(event.key).replace(/\+/g, " ");
 
   s3.getObject({
     Bucket: bucket,
     Key: key
   }, (err, data) => {
     if (err) {
       context.fail("Failed to read config file");
       return;
     }
     event.data = data.Body.toString(); // attach file content to event
     const iot = new AWS.Iot();
     iot.describeEndpoint({}, (err, data) => {
       if (err) {
         console.log(err);
         return;
       }
       event.endpoint = data.endpointAddress;
       processText(event, context);
     });
   });
 });

After this is complete, the processText(Object, Object) function will be called. It will ingest your device sample data with parseData(Object, Integer) and then set up one MQTT client per device to be simulated. This is where you can adjust the way in which the MQTT clients are created.

 function processText(params, context) {
   const mqttController = new mqtt.ClientControllerCache();
   const jsonData = parseData(params, params.numDevice);
   for (var i = 0; i < params.numDevice; i++) {
     var connectOpts = {
       accessKey: params.accessKey,
       clientId: `${Math.random().toString(36).substring(2,12)}`, // 10-bit random string
       endpoint: params.endpoint,
       secretKey: params.secretKey,
       sessionToken: params.sessionToken,
       regionName: params.region,
       topic: params.topic
     };
     var simOpts = {
       simTime: params.simTime,
       interval: params.interval,
       index: i
     };
     createMqttClient(connectOpts, simOpts, mqttController, jsonData, context);
   }
 }

The data parsing function, parseData(Object, Integer), reads in your sample data file and splits by new lines and then by comma characters. (This is the standard way of reading in a CSV file.) If you have a different file format to parse, you can edit the way this function works. To read in JSON, you’ll want to iterate over keys or a JSON array value.

 function parseData(params, numDevice) {
   var dataJSON = [];
   const lines = params.data.split('\n');
   var lineNumber = lines.length;
   for (var i = 0; i < lineNumber; i++) {
     var columns = lines[i].trim().split(',');
     var dev = [];
     for (var j = 0; j < numDevice; j++) { 
       var clientId = 'client_' + j + '@' + params.endpoint;
       dev.push({
         clientId: clientId,
         field: columns[j]
       });
     }
     dataJSON.push(dev);
   }
   return dataJSON;
 }

After parsing your data, the next step is to create one MQTT client per device to be simulated. This Lambda function uses MQTT over the WebSocket protocol to connect and publish messages to AWS IoT. The createMqttClient(Object, Object, Object, Array, Object) function sets up event handlers for the MQTT protocol events.

 function createMqttClient(connectOpts, simOpts, mqttController, jsonData, context) {
   var cbs = {
     onConnect: onConnect,
     onSubSuccess: onSubSuccess,
     onMessageArrived: onMessageArrived,
     onConnectionLost: onConnectionLost
   };
   var clientController = mqttController.getClient(connectOpts, cbs);
   /* ... define callback functions ... */
 }

The onConnect() function defines what happens when each device simulator completes the connection to AWS IoT. In this case, it then subscribes to topic, as defined in the Lambda event input (for example, test/topic). The onSubSuccess() function defines what happens when the subscription to a topic is successful. In this case, it starts the JavaScript interval, which will start publishing messages from your sample data.

 function onConnect() {
   clientController.subscribe();
 }
 function onMessageArrived(data) {
   // do nothing
 }
 function onSubSuccess() {
   var index = 0;
   var interval = setInterval(() => {
     var line = jsonData[index++][simOpts.index];
     clientController.publish(line);
   }, simOpts.interval);
   setTimeout(() => {
     clearInterval(interval);
     clientController.disconnect();
     setTimeout(() => { // set drain time to disconnect all connections
       context.succeed();
     }, 1000);
   }, simOpts.simTime);
 }
 function onConnectionLost() {
   console.log('Connection is lost');
 }

The simulator Lambda function is designed to simulate sensing devices; it does not define behavior for responding to incoming messages. If you’d like to define an actuator device or a bi-directional sensing/actuating device, we recommend that you add simulation logic to the onMessageArrived() function. By expanding the scope of topics subscribed and behaviors to take when new messages arrive, you can quickly define a device simulator that sends and receives messages.

Adding Hooks

The simplest way to start the device simulator is to send test events from the AWS Lambda console. If you’re wondering how to spin up a fleet of different devices or dynamically start simulators, we’ll describe ways to invoke Lambda functions.

One great way to define interfaces to Lambda is with Amazon API Gateway. With API Gateway, you can write an HTTP or HTTPS interface to invoke your Lambda functions. By wrapping your device simulator, you can write a web service to spin up device simulators either with a preconfigured simulator template or by passing HTTP parameters to dynamically define devices. This method makes it possible for web application and service developers to build on top of your simulator pattern. Now you can have a web hook, IFTTT, or Zapier plugin to start simulators!

Another pattern to start simulators is on a schedule with Amazon CloudWatch Events. You can specify a Cron expression that will set up a device fleet at certain times and days. Although you can add a CloudWatch event source in the AWS Lambda console, for this task the best practice is to add it from the CloudWatch console. It’s easier to configure the Lambda event input required to start your simulator. Alternatively, you can create additional defaults in your Lambda function to avoid passing in any parameters.

A third way to invoke your device simulators is in response to a message published to AWS IoT. You can set up a rule in AWS IoT that invokes your Lambda function to start up a simulated device fleet. Alternatively, you can model a fleet of devices based on a single prototype. Store all of a real device’s messages in an Amazon DynamoDB table with one rule, and then use any Lambda invocation method to replay your real device as a simulated fleet.

Finally, what better way to test your solution architecture than by running a fleet of simulated devices against it to see where it can be improved? Build a step into your workflow with Amazon Simple Workflow Service that starts up a simulation fleet as part of your flow. Or, as part of a code deployment in AWS CodeDeploy, you can send a deployment notification to an Amazon Simple Notification Service topic, which then invokes your Lambda simulator fleet to kick off a job that tests your new version of software.

Summary

This blog post provides you with a CloudFormation template for getting started with device simulation in AWS IoT and AWS Lambda. It steps you through running the template and starting the simulator. It also gives you ideas for editing, extending, and invoking the simulator. Feel free to leave comments here or share your stories about device simulation and let us know what you build in the AWS IoT forums!