Software-defined Vehicles, GenAI, IoT – The Path to AI-Defined Vehicles

The convergence of Software-defined Vehicle (SDV), Generative Artificial Intelligence (GenAI), and the Internet of Things (IoT) can pave the way for AI-defined vehicles as the future of mobility. Imagine a car that doesn’t just drive itself – it evolves with every update. In the next decade, your vehicle could be smarter, safer, and more personalized than ever before.

With SDV, Original Equipment Manufacturers (OEMs) can enhance features throughout the vehicle lifecycle, using software updates. For instance, OEMs could introduce the option for drivers to purchase new features, such as Parking Assistance or Lane Keeping. Alternatively, they can reduce the time it takes for the vehicle to reach 60 MPH through a software update. As OEMs race to implement enhanced features and capabilities, they face a critical challenge: how to effectively communicate these continuous improvements to vehicle owners.

Traditional methods of informing drivers about new features or updates—whether through printed manuals, emails, or mobile notifications—often fall short. Drivers are left unaware of valuable capabilities or struggle to access information when they need it most, especially while operating their vehicles. This communication gap threatens to undermine the very promise of SDV: seamless evolution and enhancement of the driving experience.

This is where the convergence of GenAI and IoT presents a game-changing opportunity. By reimagining how drivers interact with their vehicles’ growing capabilities, we can bridge this critical information gap and help unlock the full potential of SDV. The solution that we present here in this blog post involves storing a Small Language Model (SLM) locally in the vehicle, which can help power a voice assistant for drivers to inquire about vehicle features using natural language. This GenAI application uses the SLM to answer vehicle occupant’s questions, essentially functioning as a more readily available vehicle owner’s manual. We’ll demonstrate how the SLM lifecycle management can be handled by using and adopting SDV methodologies.

With the owner’s guide accessible through the convenience of voice-based interaction, vehicle occupants can get information about all vehicle features and operations instantly. It can also potentially increase their engagement with the vehicle brand.

In this blog post, we’ll delve into the details of implementing a voice enabled vehicle user guide and its lifecycle management, using a demo application. The blog post will also discuss virtualized development and testing concepts that can help increase productivity of the OEM teams to release vehicle user guide updates.

Solution Architecture

The diagram in Figure 1 illustrates the solution architecture. We’ll discuss the design and implementation details in the following sections.

Figure 1: Solution architecture diagram for GenAI based vehicle user guide

Creating the GenAI Model

Choosing a Base Model
Our approach to creating a GenAI model involves selecting a suitable base model for fine-tuning using vehicle-specific user guide data. For a vehicle user guide chat assistant, we prioritized a model pre-trained for conversational tasks.

For this demo, we chose NXP’s i.MX 8M Plus as the target device, which has 5.5 GB of memory. In the context of this i.MX 8M Plus target device, we need to choose a model that will fit within the 5.5 GB memory. We shortlisted TinyLlama-1.1B-Chat-v1.0 for its robustness in chatting and compact size. Please note that choosing a base model from specialized vendors can provide an automotive-domain specific starting point.

With a file size of 2.2 GB, TinyLlama-1.1B-Chat-v1.0 file was still large for the target device and over-the-air deployment. This is where model quantization proved valuable. As described on the Hugging Face website, “Quantization reduces computational and memory costs by representing weights and activations with low-precision data types”. We used the q4_K_M quantization method, which reduced the file size of the model to 637 MB.

We successfully validated the model’s usability on the target device, using a virtual NXP i.MX 8M plus device on Arm Virtual Hardware (AVH). We have provided more details about AVH in the subsequent section titled Virtual Hardware. After validating compatibility with the target device, we decided to use TinyLlama-1.1B-Chat-v1.0 as the model.

Creating Quality Data

Creating high-quality, comprehensive training data is crucial for accurate results from the fine-tuned model. For our vehicle user guide chat interface, we designed the chat interface to accommodate the driver’s focus on operating the vehicle, by crafting succinct answers. We created a new dataset from scratch, containing 1000 rows in question-and-answer format. This 1,000-row dataset significantly improved response quality and accuracy compared to a smaller 200-row dataset. The complete dataset is available in the code repository. OEMs can consider using their existing vehicle user guide as the starting point to create curated content tailored for the GenAI assistant use case.

Fine-tuning the Model

We used Amazon SageMaker Studio for model fine-tuning. Fine-tuning a pre-trained model can be cost prohibitive. To help with this, we used the Low-Rank Adaptation (LoRA) method for Parameter-Efficient Fine-Tuning (PEFT). With LoRA, it is important to select the right parameters to produce a high-quality fine-tuned model. Referencing this page, we selected the key parameter values.

The source model’s Model card page includes a prompt template. To get quality responses from the model, it is important to utilize this prompt template and create a prompt specific to the vehicle user guide use case. We have utilized it and created a prompt, which you can see in the Notebook file. Instructions to run the notebook using SageMaker Studio is available in the code repository’s README.

The fine-tuned model, with a file size of 3.8 GB and a test loss of 0.15, was stored in a S3 bucket.

SDV Development and Release stages

Using the approach shown in Figure 2 helps accelerate the development and release of SDV components by the OEM.

Figure 2: SDV Development and Release stages

This approach helps provides time to market advantage, allowing development and testing on AWS and Virtual Hardware environments without waiting for the physical hardware. This approach also helps to identify and rectify code issues early in the development process, which is important for mixed-critical AI applications. The following sections will explain each of these stages in the context of the GenAI assistant use case.

1. AWS AMI

Creating the Chat Application
This section discusses developing the chat application, with potential for future voice-based interaction.

Reducing Model File Size
To optimize the 3.8 GB fine-tuned model for edge device processing, we used an Amazon EC2 (EC2) instance running eLxr OS. eLxr OS was obtained using the AMI from AWS Marketplace.

eLxr is a fully open source, Debian-based OS optimized for fast boot times and high performance on edge devices. eLxr is available for multiple platforms and targets. For the development of this demo, we found eLxr’s availability, stability and performance to be suitable across the physical NXP i.MX 8M plus device, the AVH based virtual NXP i.MX 8M plus device, and on the EC2 instance.

We then used Ollama for model quantization, applying the q4_K_M method. The quantized model size was reduced from 3.8 GB to 637 MB. When asked a question from the curated dataset, the fine-tuned model provided the correct answer 83% of the time. The accuracy can be further improved by fine-tuning the model with additional data.

You can experiment with other available quantization methods.

Python-based Chat Application
On the same EC2 instance, we developed a Flask-based Python service, exposing a RESTful API for chatbot interactions. The application uses llama-cpp-python to load the quantized model and generate responses to input queries.

2. Virtual Hardware

AVH provides access to many popular IoT development kits. Developing and testing on AVH provides time savings to organizations when the actual physical device is not available, or inaccessible by globally distributed teams. We successfully tested the demo application on AVH, where AVH provided a virtual instance of the NXP i.MX 8M plus device. Please refer to the README in the code repository for the further information.

3. Physical Hardware

For running the application on the physical hardware at the edge, we used AWS IoT Greengrass, an open-source edge runtime and cloud service for building, deploying, and managing device software. Greengrass APIs can package the application and deploy them over-the-air to the physical device. We used the standard device configuration and package deployment instructions available in the Greengrass documentation. This demo utilized Greengrass to deliver the application artifacts over-the-air to the physical hardware. Production usage may combine additional software to deliver the updates to the target device inside the vehicle.

The Big Loop – Continuous fine-tuning of the model

Beyond in-house testing, collecting model accuracy feedback from the vehicle fleet can help further fine-tune the model. Deploying the updated model back to the vehicles will complete the big loop. We need to identify an approach to get the model accuracy feedback potentially without distracting the vehicle driver. One approach is to measure the distance between the answer that the model chose, and the answer that was used in the fine-tuning. If the distance exceeds the established threshold, such questions can be sent back automatically by the GenAI application to AWS for model refinement. Greengrass can be used by the OEM to send the feedback back into AWS. We will elaborate on this distance measure approach in a follow-up blog.

Conclusion

The integration of SDV, GenAI, and IoT is helping transform the automotive industry. Our demonstration of a GenAI-powered in-vehicle assistant proves how these technologies can help create smarter, more engaging vehicles. Our project showcases innovations in edge-deployed GenAI models, real-time IoT updates, and virtualized testing. We discovered that successful implementation requires careful model selection and targeted data curation. Training and inference across different hardware platforms present unique challenges. Regular updates are essential to keep the model relevant to evolving SDV capabilities. These advancements mark the beginning of AI-defined vehicles, promising drivers a more personalized and interactive experience. We look forward to sharing more insights as this technology evolves.

You can explore the demo code in our repository as well as watch the re:Invent 2024 presentation on this topic here. Stay tuned for future blogs on the AI-defined vehicle topic.