How to solve the digital twin challenge using building blocks from MongoDB on AWS

In the first blog – Digital Twin Data Middleware with AWS and MongoDB – we discussed the business implications of the digital twin challenge and how MongoDB and AWS are well positioned to solve them.

In this blog, we’ll dive into technical aspects of solving the digital twin challenge. That is, showing you how MongoDB and AWS provide you with building blocks for efficient bidirectional integration between mobile devices, databases, the cloud and vehicle, as shown in Figure 1.

Figure 1: High Level Overview

The Connected Vehicle Platform Environment

By 2025, an estimated 400 million connected vehicles are expected to be on the road.

In order to provide new and innovative experiences for customers through the integration of connected vehicles, there needs to be seamless communication and synchronization of information between vehicles, mobile applications and cloud services. This is a critical component, but also challenging to develop and improve.

Using MongoDB Atlas with the AWS ecosystem help solve this challenge and allow you to build new personalized customer experiences with ease. We walk you through each step to help you build a connected vehicle environment as depicted in Figure 2:

Step 1: Setting up the MongoDB Atlas Cloud Backend
Step 2: Setting up the AWS Ecosystem Integration

Figure 2: High Level Architecture of a Connected Vehicle Environment

Now let’s have a look at the first building block required which is setting up the MongoDB Atlas cloud backend.

Setting up the MongoDB Atlas Cloud Backend

Figure 3: MongoDB Atlas Backend

MongoDB Atlas is a developer data platform and the perfect fit for developers looking to build the next generation of connected vehicles platform. With its robust cloud infrastructure powered by AWS including features such as globally distributed clusters, Atlas Triggers, Device Sync, Authentication, and GraphQL API endpoints, you can quickly get started on creating a connected vehicle environment.

For a hands on deep dive where you can learn how to set up your MongoDB Atlas Digital-Twin Backend, please refer to Part 1 of the Digital-Twin GitHub repository.

But for now, let’s jump into the Atlas capabilities you’ll use.

Atlas Data Services – The MongoDB Database Cluster

Once your MongoDB Atlas account is set up, you can spin up a MongoDB database cluster in a few clicks. This will act as the persistence layer for all your synchronized vehicle data. MongoDB databases use the document model (JSON structures) as the data model. The document model is the superset of all other data models and is natural to software developers as it maps directly to objects in code. Its flexibility also allows efficient storage and processing of a wide variety of data from digital twin context information to alerts and telemetry data. All of these can be processed through a single API with the MongoDB query language “MQL”.

Because there are millions of connected vehicles on the road worldwide, OEMs must also ensure scalability and keep control over data distribution, to accommodate regulatory requirements. MongoDB database clusters easily accommodate large volumes of data due to the distributed architecture and horizontal scalability.

Figure 4: MongoDB Database Architecture

As shown in Figure 4, replica sets consist of one primary and up to 50 secondary nodes, guaranteeing high availability and scale-out for read/analytical workloads. To scale writes and accommodate data locality requirements, data sets can be split up into multiple shards with full control over where each document resides. A sharded cluster requires query router(s) called “mongos” to target queries to distinct nodes. Nodes and shards can be added or removed transparently to the application while the cluster is fully operational to adapt to growth/decline.

Atlas App Services – Serverless Backend Services

Now that we have covered the MongoDB database cluster, we want to create a data middleware between the vehicle, mobile apps and the cloud backend. This seamless data layer will allow the efficient sharing of information like vehicle state, telemetry, vehicle settings, and much more based on your requirements. For this purpose, we are going to leverage the different capabilities of Atlas App Services, a fully-managed serverless backend that helps you build data-driven apps, integrate services, and innovate faster.

Realm SDKs and serverless services such as Device Sync, Triggers, and GraphQL or the Data API make it easy to build and connect a wide variety of devices, applications or microservices to MongoDB Atlas.

Atlas App Services allows developers to create multiple applications, each with their own settings and configurations. These applications can be maintained via command line utility or through direct GitHub integration. You can go ahead and create your own application from scratch (get Started) or directly import our preconfigured Digital Twin Backend application.

Now let’s see how each of these components help you build a Digital Twin data middleware.

Atlas App Services – Authentication / Authorization

The first service we are looking into is authentication and authorization. A lot of data within connected vehicles is sensitive and of very private nature. Therefore, you want to make sure you have the right control mechanisms available to control who can access which piece of information down to the field level. Atlas App Services makes this easy by allowing you to seamlessly leverage popular authentication providers such as Google, Apple and Facebook or integrate with identity providers such as Amazon Cognito.

For our sample scenario we decided to leverage the Email/Password authentication provider, which allows you to manage usernames and passwords directly inside of MongoDB Atlas. Once a user is authenticated, the right set of permissions have to be applied. Atlas App Services allows you to create access rules down to the field level depending on specific user attributes directly provided by the authentication provider or stored as part of Custom User Data.

Figure 5: Sample Access Rule

Atlas App Services – Realm SDKs / Device Sync

For our Digital Twin use case, we have built two separate applications leveraging two of our different Realm SDKs. The vehicle simulator uses the JavaScript/TypeScript SDK and the vehicle controller mobile app uses popular Swift SDK. The Realm SDKs contain the hugely popular Realm embedded database. This provides a sophisticated object-oriented API to persist objects directly to the file system with no need for additional frameworks such as ORMs. Vehicle settings or user profiles can therefore be stored within the vehicle or on the mobile device following the offline first paradigm even though connectivity isn’t available. This is done by collecting and storing telemetry data for later transmission to the cloud backend or processing directly inside the vehicle.

Figure 6: End-to-End Overview

Realm SDKs use Atlas Device Sync to share data via the cloud backend with other applications also leveraging Realm SDKs. The bidirectional data synchronization happens within milliseconds, making it easy to share for example a real time synchronized representation of your vehicle state with your mobile app.

Wireless networks in remote locations can be unreliable with low bandwidth or expensive roaming costs attached. Atlas Device Sync reduces the amount of transferred data as much as possible by only synchronizing the changes/deltas and applying compression transparently before sending data across the network. To make the life of a software developer even easier, the synchronization protocol also provides built-in deterministic conflict resolution for scenarios where multiple changes to the data have occurred while devices were offline and now need to be merged again.

For in depth step-by-step instructions on how to build the two apps, please refer to Part 2 and Part 3 of our GitHub repository.

Setting up the AWS Ecosystem Integration

Atlas App Services – Triggers / AWS Ecosystem Integration

Innovative customer experiences require real-time reactions to changing data, such as low battery status in a vehicle. In our demo application, we collect and group battery status measurements into buckets of 20 before sending them to the cloud backend for analysis. This will optimize transmission efficiency through better compression and less overhead.

Once the data arrives in the MongoDB Atlas database, it is time for analysis. MongoDB Atlas is often paired with AI/ML systems to create intelligent apps. Live and historic data stored in MongoDB feeds ML platforms for model training, inference and observability using a set of pre-built connectors and APIs. In this particular scenario we want to forward the data to Amazon SageMaker for analysis. Atlas Triggers and the seamless integration with Amazon EventBridge serve this purpose.

Figure 7: End-to-End Overview

The configuration of Atlas Triggers is very easy and straightforward. All you have to do is define the execution criteria such as “on insert/update etc.” or create a schedule for time-based execution. Within the trigger, you can then specify a JavaScript function to be executed or as we use it in the demo app, use the seamless Amazon EventBridge integration and publish directly without the need for an additional function. This makes integrating with the powerful AWS ecosystem extremely easy and allows us to bring data to Amazon SageMaker with minimal coding and also optionally integrate with Amazon SNS for push notifications.

To get a glimpse at how such an integration can be configured, have a look at our demo application Amazon SageMaker Integration.

Once Amazon SageMaker has processed the data, the inference is returned to EventBridge from where we use a AWS Lambda function to write it back into a distinct MongoDB collection, explicitly not directly updating the synchronized data set, to allow some preprocessing such as enriching the inference with some more user-friendly localized text and actions to undertake.
As you likely imagined, this is also a great example for the use of Atlas Triggers where we wait for the insertion of the SageMaker result and then execute an Atlas Trigger to update the synchronized vehicle model with the right information for the vehicle or mobile app user.

Atlas App Services – GraphQL API

Last but not least with MongoDB and GraphQL, you can remove the dreaded SQL joins. Atlas App Services provides you with a fully managed Atlas GraphQL API including integration with the previously mentioned authentication providers and role-based permissions down to field level. The beauty of GraphQL is that it goes nicely along with the document model and allows you to seamlessly query hierarchical data across multiple MongoDB collections efficiently and without over fetching. The schema required for GraphQL is automatically synchronized with the JSON schema used for permissions and the class definitions for the different SDKs, making the whole software development process an efficient and frictionless experience.

If you have deployed the Digital Twin demo application, you can simply click on your Vehicle schema in Atlas and explore the JSON schema defined for the vehicle and its components or have a look at Figure 8 below, where you also see how relationships between objects in different collections can be defined.

Figure 8: JSON Schema 

MongoDB Atlas App Services also provides a simple user interface to run and test GraphQL queries directly against your data.

Figure 9: Query your data with GraphQL

While GraphQL is great for querying hierarchical datasets, writing and updating data through GraphQL can be quite difficult and take some unnecessary development time. To make working with data as easy as with the MongoDB drivers but via REST API, we provide the Atlas Data API.
Similar to the GraphQL API you can enable the Data API with a single click. You are provided a https endpoint over which you access your data inside the MongoDB cluster based upon your assigned role by the authentication provider configured. All you have to do is authenticate and post your “MQL” query or aggregation pipeline and expect the result.

Conclusion

In this blog we dived deeper into the technical aspects of solving the Digital Twin Challenge of creating seamless bidirectional integration with MongoDB and AWS. However the best way to learn new technologies and concepts is hands-on, so we recommend going through the GitHub repository thoroughly step by step and running the demo environment.

If you’re interested to learn more about how you can use MongoDB to drive the next-generation of customer experiences and seamlessly solve Digital Twin Challenges, feel free to connect with: industry.solutions@mongodb.com