Five’s Automated Driving Systems (ADS) Testing Platform on AWS

Five was founded in Cambridge, England in 2015 to build a complete autonomous vehicle system. Its team of computer scientists, engineers, and roboticists built a Level 4 urban automated driving system which was demonstrated in London in 2019.

Five took the lessons from that work and created a platform for the development and safety assurance of advanced driver assistance systems (ADAS) and automated driving systems (ADS). The platform includes the functionality to create and choose scenarios, run varied simulations, evaluate self-driving model performance, and refine a self-driving stack against data captured from the real world. The platform is designed to help companies advance through Society of Automotive Engineers (SAE) J3016 Level 3 and 4 autonomy challenges from navigating traffic jams to highway driving and, ultimately, to complex urban environments

Five selected AWS on which to build this platform because it enables the dynamic storage and processing petabytes (PB) of data, access to the largest scale of the latest compute technology, and a broad suite of tools to integrate on-premises infrastructure, providing a seamless edge-to-cloud environment ideal for ADS development.

In this blog, we review Five’s approach to ensuring ADS safety using a novel, model-based development workflow to explore the performance of an ADS at massive scale in a highly integrated, cloud-native platform custom built on AWS.

 “AWS has extensive offerings in access policy definitions and cross-account data access has made it possible for us to deliver PaaS-like Automated Driving Systems (ADS)  solutions that don’t require our partners to give up data ownership or replicate data across AWS environments.” – Stan Boland, CEO – Five AI

Arguing Safety of a Self-Driving System

A number of industry bodies are working to develop standardized processes aimed at arguing the safety performance of ADAS and autonomous vehicle systems. ISO is one such body and its 26262 standard has been widely adopted by the automotive industry to reduce the risk of software bugs and hardware failures with the assumption that requirements for these systems can be defined comprehensively. However, ISO 26262 was never intended to address the potential for specification defects.

ISO 21448 was introduced to address the problem of whether the specified behavior is appropriate in all potential operating conditions across the target Operational Design Domain (ODD). Known as Safety of the Intended Functionality, or “SOTIF”, the standard includes a means of assessing system hazards, risks, performance, and failures due to limitations of algorithms, models, noise, sensor performance, and many other factors. This is a difficult task. Developers must evaluate the self-driving system performance over sufficient combinations of all ODD parameters to be statistically confident of reaching a minimal tolerable level of safety in the domain.

ISO/TR 4804 “Road Vehicles – safety and cybersecurity for automated driving systems – design, verification, and validation” was published in 2020 and is based on the “Safety First for Automated Driving” (SaFAD) whitepaper authored by various industry leaders. This standard promotes a statistical demonstration of safety using a combination of the following three strategies:

• Statistical gray box testing in real-world driving tests to cover the variety of real-world driving scenarios
• Scenario-based testing for the complete driving system in addition to specific elements in dedicated test platforms
• Field monitoring of the system over its lifetime

At the heart of this challenge for safety assurance is the complex interdependence between the perception and planning systems, as Five learned firsthand from building our own self-driving system. As discussed in SaFAD, it’s straightforward to evaluate whether a new version of a perception system has improved on the errors that led to a real-world drive disengagement. However, the new version of the perception system will impact the planning and prediction system that depends on it. We can prove the ADS won’t fail in precisely the way it has done in the past, but this approach alone does not give proof that it won’t fail in some new way.

A simulation environment that can interact with a system under test (SUT) and accurately represent the ODD is required.  Five’s approach included building a photo-real simulator, which has been useful for finding bugs that manifest only when the full-stack is integrated and running on target hardware. The biggest drawback of photorealistic simulation is that it does not replicate the perception errors that we find in the real-world.

Figure 1. A cut-in scenario being created in the Five Scenario Editor. Scenarios are defined using ‘interactions’ between traffic participants, which enables robustness against variations in the ADS behavior.

Five has pioneered an alternative approach as a pathway to arguing the statistical safety of a system based on exploring the performance of an ADS against abstractions of real-world ODD characteristics into accurate models that can be efficiently explored at scale.

The first of these are abstract, robust scenarios, for which Five has built an easy-to-use scenario editor. The Five scenario format is independent of concrete maps and defined relative to an ODD. This means that the same ‘conceptual’ scenario could be implemented on any suitable road geometry within the customer ODD. In particular, maps of different geographical areas can be used and different ODD parameters, such as acceptable precipitation or lighting, can be supplied.

Five’s platform also includes the tools to create, parameterize, validate, sample from, and use models of the error of the perception ADS-under-test.  We refer to these as parametric error models (PEM) which provide a method of sampling statistically representative outputs of perception systems at scale and at low cost in a way, which will speed up the detection of system-level errors and provide relevant evidence for safety arguments.

PEMs can be fitted to the real errors observed in the system over pose, velocity, track creation/destruction, and false positives/negatives. The ground truth data for training the error models and characterizing the errors of the perception system can be efficiently provided by Five’s offline perception refinement processing pipeline coupled with an established relationship with our labeling partner.

Vast parameter combinations of these scenarios are explored in combination with the statistically accurate perception error characteristics using low-cost, massively scalable ‘headless’ simulation. For the first time, this means that system performance can be explored efficiently within hyperscale simulation that is representative of the real-world.

Drawing actionable analysis from these vast simulations is the next step, for which Five has developed its Test Oracle framework. This framework evaluates a customer-specified set of rules against every timestamp in each scenario played in order to assess the performance of the ADS.  These road rules are written in a highly expressive Domain Specific Language (DSL) in the Five Platform Rule Editor tool and deployed to a database of named rulesets.  Once deployed, rules can immediately be part of a ruleset for running a test suite.  The Test Oracle is designed to provide a flexible testing tool that can be tailored to different safety models and scenarios with minimal effort.

Figure 2. The Five Test Oracle evaluating codified Automated Lane Keeping Systems (ALKS) regulation rules on a highway cut-in scenario.

Building for Scale and Complexity: AWS as the Partner of Choice

Data is at the core of Five’s ADS safety assurance platform. AWS offers a number of ways to bring real world data from the edge to the cloud including 10 Gigabit-per-second (Gbps) and 100 Gbps AWS Direct Connect (DX) links. For storage, AWS Managed Services, Amazon S3 and Amazon RDS, enable scaling of data collection and simulation pipelines in an ADS data lake without having to worry about provisioning the underlying storage infrastructure. Amazon S3 also offers the functionality to automate storage lifecycle policies with S3 Intelligent-Tiering. Amazon S3 Glacier and S3 Glacier Deep Archive offer Five the ability to cost effectively manage a vast data lake at minimal cost without having to consider removing potentially valuable sensor data.

Five offers highly scalable SIL-based simulation and scenario evaluation pipelines, made possible by AWS services, without limiting the ability to integrate closely with on-premises HIL benches. Five’s services for real-world sensor data pipelines utilize AWS services to simplify data storage and data sharing, making it possible to offer highly integrated services without their partners being required to give up data ownership

Delivering Five’s platform entirely as a web-based software as a service (SaaS) would be impractical given its complexity and scale. Customer data is fundamental to many of Five’s solutions and accessing, integrating, and processing that data at PB-scale is a non-trivial problem. AWS has extensive offerings in policy definitions and cross-account data access, which has made it possible for Five to deliver a platform as a service (PaaS)-like ADS solution without requiring our partners to give up ownership of their data or replicate data across different AWS environments.

For compute infrastructure, the Five platform relies on Amazon EC2 and Amazon Elastic Kubernetes Service (EKS) for container scheduling and orchestration. AWS Organizations integration with kubernetes permission structures,  AWS Identity and Access Management (IAM) roles, and Five’s single sign-on provider, Okta, means that Five is able to deliver self-service developer environments which closely resemble production clusters. This setup allows Five’s customers to develop faster, in a more structured way, and adopt automated deployment procedures to allow faster iteration. It also allows Five’s platform to deliver reliable and secure ADS evaluation in a sandboxed environment.

Five uses a number of EC2 instances including M5, C5 , R5, and P3 and pools them in Auto Scaling groups to automatically scale-out and scale-in compute nodes as the application requires. EC2 Spot Instances are core to this platform as they enable cost savings of up to 90% versus on-demand instances. The close integration between EKS, EC2 spot, and automatic scaling makes it possible to both horizontally and vertically scale a compute cluster depending upon the specific computing profile of a given pool of simulations. Five can incorporate both CPU, GPU, and other accelerator-based compute thanks to the 17 distinct EC2 instance families AWS offers.

Conclusion

This blog describes Five’s unique approach to ADS safety validation by accounting for the impact of updated perception models on planning algorithms. Five’s platform uses a wide variety of parameter combinations where perception systems are surrogated using PEMs and are leveraged to argue statistical safety, negating the need for complex ray tracing and sensor modeling. Five’s platform is customizable to specific ADS testing requirements and can be deployed in any AWS environment. Five and AWS take a consultative approach to fully understand customer ADS development workflows and can architect a bespoke, cloud native environment that seamlessly integrates with real world, on-premises infrastructure. Reach out to info@five.ai for an introductory demonstration.

About Five.ai

Five is Europe’s leading automotive technology company. Based in Cambridge, UK, Five was founded in 2015 to build a complete autonomous vehicle system that would free people to pursue their dreams without the need to drive.

Its team of computer scientists, engineers, and roboticists has built a high functioning, Level 4 urban automated driving system prototype, first demonstrated on the streets of London in the StreetWise trials of September 2019.

More recently, Five pivoted to building a highly integrated cloud-native platform of workflows for the development, verification, and validation of advanced ADS for deployment by automotive manufacturers and automotive component suppliers for use in the development of next generation vehicles.

Five’s workflows include scenario building, simulation, coverage tools, and performance evaluation through behavioral rulesets, all of which leverage Five’s experience building an advanced ADS.