How INESC TEC tests and deploys ROS applications with AWS RoboMaker

This blog was co-authored by Rafael Arrais, Project Manager, and Sergio Teixeira, Researcher, at INESCTEC.

Introduction

The Institute for Systems and Computer Engineering, Technology, and Science (INESC TEC) is a research and development institute headquartered at the University of Porto in Portugal. In this blog, we discuss how robotics researchers at INESC TEC automate testing and deployment of their Robot Operating System (ROS)-application in AWS RoboMaker simulation.

The team at INESC TEC coordinates the H2020 FASTEN Project, a research and innovation effort co-funded by the European Commission and the Brazilian government to develop, demonstrate, validate, and disseminate an integrated and modular framework for efficiently manufacturing custom-designed products. The INESC TEC researchers worked on provisioning robotics software for a mobile manipulator, responsible for performing automatic pick and place tasks in material handling stations, interfacing with warehouse equipment and operators. The goal of this pilot, conducted with Embraer Portugal S.A., was to demonstrate that automation of these tasks can lead to a faster, more flexible, and reliable supply of components to the production line.

Challenges to Scale and Automate Testing

We faced three main challenges to scale and automate testing.

First, we needed robot behavior testing to catch and fix potential bugs in the robotics software. Testing robotics software to ensure a robot will behave as expected in a dynamic real-world environment is a complicated and time-consuming task. Initially, we tried to do this testing by using real robots either in the laboratory or in industrial environments. This led to logistical scheduling issues, robot downtime, and human resource allocation costs. In reality, it was generally impractical to test each code committal with the same level of repeatability, since it required the full deployment to a real robot, resulting in significant delay and associated costs. As a result, we were only able to test code once every week or every two weeks. To avoid these issues, we leveraged simulation-based testing to virtually catch and fix robot behavior bugs in the Gazebo simulation software, before deploying to expensive physical hardware. This type of testing would not be possible purely through software-only unit and integration tests.

Second, we needed to scale up the number of simulations running, in order to test the code base often and more robustly. We were limited by the type and number of test scenarios we could run on our on-premises systems. With our existing simulation infrastructure, it became expensive and time consuming to efficiently simulate different combinations of scenarios, covering the full robotic software stack.

Third, we wanted to shorten our release cycle to get our products to market faster without sacrificing the overall software code quality. To do this, we needed a continuous integration (CI) pipeline so developers can test their code with each code commit. Our researchers built a Travis CI and a GitLab CI pipeline for their software unit and integration tests and wanted to automatically trigger Gazebo simulation-tests as part of the CI process. We could successfully run unit, integration, and simulation tests with each code commit but with the existing on-premises infrastructure, we did not have a way to deploy new software updates to production robot, in labs around the world.

INESC TEC’s Solution

We decided to explore AWS RoboMaker for building an automated CI/CD pipeline. This provides us a fully managed robotics simulation service that we can use to run multiple, parallel, physics-based scenario tests. AWS RoboMaker enables us to quickly and efficiently run hundreds of simulations in batch to test the variety of different combinations and permutations of conditions that would exist in the real world. This solution is scalable, cloud-native and takes advantage of AWS RoboMaker’s integrations with other AWS services for building a ROS-application CI/CD pipeline. Before deciding to use AWS RoboMaker, we explored few alternatives. One alternative was to build our own cloud-based simulation infrastructure using a custom deployment of ROS and Gazebo to a Virtual Private Server (VPS) hosted in the cloud. This approach was quickly abandoned since developing such a system is not straightforward and would require a substantial allocation of the team’s resources.

Figure 1: ROS-application CI/CD Architecture

How the Solution Works

The architecture diagram for our solution is shown in the Figure1. Every time a developer makes a commit or pull request to the code base (either in GitLab Enterprise or GitHub), the entire CI/CD pipeline is triggered.

Step 1: Build and run unit tests using the ROS Industrial CI stack (industrial_ci).

Step 2: When the bundle process is completed successfully, the next step is to upload the bundles to a specified Amazon S3 bucket that should contain both the robot and simulation application bundles, overwriting any existing application.

Step 3: After the upload, the next step is launching the simulation itself. First, the pipeline uses an AWS Lambda Function, a service that lets you easily build state machines in the cloud.

The state machine is responsible for the simulation execution, monitoring, and analysis with user-defined parameters as the input. The state machine is defined by a group of previously created and immutable AWS Lambda Functions, as shown in Figure 2.

Step 1: The first state of this state machine is a Lambda Function which creates and runs an AWS RoboMaker Simulation Job using the new version of the Robot Application that was bundled and uploaded in a previous step.

Step 2: After this simulation job is created, the next step is monitoring, where we use another AWS Lambda function to check the simulation job status and whether it either failed or was successful.

Step 3: If the simulation is completed with success, the next and final state is the simulation analysis. The team wrote an AWS Lambda Function responsible to analyze the simulation logs and check whether or not it was completed according to the desired parameters.

Figure 2: Step Function used for interfacing with CI pipeline

While these steps are being handled by AWS services, a CI script is waiting for a response from the AWS Step Function’s state machine. Once this response is received, the CI process output is chosen – it passes if everything went well, and it fails the pipeline if any of the steps defined preceding failed or was not completed as expected. Assuming the commit or pull request is being done to the main branch, and assuming a successful CI run, the Continuous Deployment (CD) pipeline is completed next.

The CD pipeline consists of creating a Docker image containing the robot application and all the required dependencies based on an existing Dockerfile, hosted at the project repository (GitLab link). After this Docker image is generated, it is uploaded to the GitLab Enterprise Docker Registry as a new version of the application. So, every time a change is made to the code stack, and if the CI pipeline is completed successfully, a new Docker image of that version is generated and stored. This image is readily available for automatic deployment to the robotic fleet, using AWS RoboMaker’s fleet management over-the-air (OTA) offering.

Solution Results

Using AWS RoboMaker with other AWS services such as Amazon S3, AWS Lambda Functions, and AWS Step Functions, we were able to test code on each code commit, thus improving the quality of the codebase. In addition, AWS RoboMaker allowed us to migrate physical robot-specific testing to cloud-based simulations, fully integrated into an existing CI/CD pipeline. By automating our testing infrastructure in simulation, we were able to speed up our testing time from 4 hours per test in the physical world to a couple of minutes in RoboMaker simulation. This allowed the researchers to focus on more important and value-added operations.

By relying on cloud-based simulation, the development and consequent testing processes became automated and abstracted from issues arising from the actual physical usage of robotic systems, such as hardware availability and safety concerns. AWS allowed INESC TEC to scale the reach of their development and enable usage of their software by different, multi-disciplinary teams across the world.

Figure 3: AWS RoboMaker Simulation

Conclusion:

In this blog, we described how robotics researchers at INESC TEC in Portugal used AWS RoboMaker to set up a simulation-based regression testing CI/CD pipeline.  We walked you through how they used simulation to scale testing, speed up dev velocity and reduce errors. If you want to know more about testing robots in simulation, read our blog: Introduction to Automatic Testing of Robotics Applications and Building a ROS-Application CI Pipeline with AWS RoboMaker.

If you are interested in learning more about this application, Contact Us to find out more!