Middleware in Autonomous Vehicles

Autonomous vehicles (AV), more popularly called self-driving cars, have been a technological challenge that is yet to be fully conquered by any company. Over the last decade, several technological advancements have led to the miniaturization of hardware to an acceptable form which has led to the development and testing of Deep Learning algorithms for various sub-applications such as street sign classification, lane classification, etc. The overarching goal is full autonomy also called L5.

1. Level 0: No driving automation
2. Level 1: Driver assistance
3. Level 2: Partial driving automation
4. Level 3: Conditional driving automation
5. Level 4: High driving automation
6. Level 5: Full driving automation

Autonomous vehicle hardware development platforms, whether hardware in loop test setup or a car fitted with sensors such as cameras, LiDars, Radars, GPS, etc., often have a hybrid compute architecture. The safety software components are run on ASIL D-certified hardware, such as an Infineon Aurix microcontroller, while the Machine Learning algorithm is run on compute-intensive hardware supporting GPUs/NPUs, etc. A software layer sandwiched between the operating system and algorithm applications is called middleware.

The acquisition of sensor data and presenting it to multiple algorithm applications, acquiring control output from the algorithm, and pushing the data to the safety component which is the gateway to the car's ECU (Embedded Control Unit) is what middleware does.

When the first development of self-driving cars started, researchers often used a middleware framework called Robot Operating System, or ROS. It provided a publisher-subscriber architecture allowing rapid prototyping. While in theory, this advanced the algorithms, the development missed the aspect of safety certification. Each electronic component and its software had to meet a standard, e.g., ISO 26262. From the TUVSUD website:

"ISO 26262 is an international standard for functional safety in the automotive industry. The standard applies to electrical and electronic systems consisting of hardware and software components in vehicles. ISO 26262 defines requirements to be met by the safety relevant function of the system as well as by processes, methods and tools which are used within the development process. The ISO 26262 standard ensures that sufficient levels of safety are being met and maintained throughout the vehicle lifecycle."

Figure 1: V-model

Figure 1 shows a V-Model, also called the verification and validation model. The model is prescribed by international industry standards for the development of safety-critical systems like the ISO26262. While the left side of the V model focuses on requirements at both the macro and micro levels, the right side focuses on integration and testing. Variations of this V-Model are employed in both hardware and software development for automotive products.

Adaptive AUTOSAR

Traditional automakers relied on the AUTOSAR standard for hardware and software development. AUTOSAR standard is maintained by a consortium of hundreds of companies related to automobiles. While the AUTOSAR standard incorporated elements from ISO26262, it does not guarantee meeting the ISO 26262 criteria. However, companies such as Vector and Elektrobit, part of the AUTOSAR consortium, have their own AUTOSAR-based framework meeting the ISO 26262 standards.

The AUTOSAR framework evolved over time and has a couple of flavors based on the use cases Classic and Adaptive. The Classic framework focused on Embedded ECUs (Engine Control Units) with hard real-time constraints and redundancy. They were used to replace or support electro-mechanical systems. The software takes input signals and controls the actuator with an output signal. The input and output signals seldom change over the course of a vehicle's life. The Adaptive platform was developed to tackle new challenges related to connectivity, autonomous driving, etc. It uses compute-intensive hardware which may evolve over time thus requiring change in both hardware and software.

Figure 2: Adaptive AUTOSAR

Figure 2 shows the component of the Adaptive AUTOSAR Framework. The applications run on top of the run time environment which provides C++ APIs from the platform-level function blocks. 

Operating System and ExecutionManager/HealthManager

When the system boots, the Execution Manager (EM), typically a daemon, is automatically started by the OS. The entire application life cycle is managed by the Execution Manager based on a static or dynamic configuration of applications. From the EM's point of view, all other functional blocks and services are applications whose life-cycle it controls. It is responsible for the deterministic execution of the applications. Along with the Execution Manager, another sub-module is the Health Manager (HM). It is often packaged along with the EM and has an interface to the hardware watchdog. In the event of a misbehaving application, the HM can trigger a watchdog panic or reset the system. The Operating System Function block provides a subset of the POSIX interface called PSE51 to allow for portability with different underlying OS.

The PSE51 standard does not support IPC, and therefore, all communication between different applications uses the Communication Manager. Since applications could potentially run on a remote ECU system, the communication manager provides the applications with seamless integration and communication with each other. The Communication Manager (CM) is an integral aspect of the data movement pipeline required in Autonomous Driving and we will take a deeper look into it next.

Communication Manager and REST

The CM abstracts the complexities of creating the data pipeline in a distributed system. AUTOSAR provides two communication sub-function blocks: the traditional SOA-based architecture as well as the REST-based communication architecture. AUTOSAR REST comprises of a URI, request methods such as HTTP or POST, and an object graph data structure that maps a protocol format between JSON or C-like structs. REST has network endpoints for Server and Client communication.

Based on the notion of Service Oriented Architecture (SOA), SOA (Figure 3) postulates that a system consists of services that other applications may interact with. An application is both a service and a consumer of the service. AUTOSAR employs the SOME/IP as the communication backbone which uses the IP protocols like TCP/UDP.

Figure 3: Service Oriented Architecture

The API design uses the proxy/skeleton pattern. The APIs are defined in a service contract in an AUTOSAR XML (ARXML) file, which is used to generate C++ stubs for proxy and service classes to be consumed by the client application and service implementation, as shown in Figure 4.

Figure 4: Proxy-Skeleton Architecture

The generated stubs have support for a zero-copy mechanism for memory management. It has mechanisms to filter the data received, call-back mechanisms, and event and polling styles of API usage.

Amongst others, the AUTOSAR model supports the following message types:

1. Events: A unidirectional message sent from the producer service to the consumer application to notify of the occurrence of a specific event
2. Fields: A type of message that has a status and a value that the consumers can use by utilizing getter and setter APIs or a notifier-callback mechanism
3. Method: It is a function or sub-routine invoked by the client and handled by the service application.

Depending on the service contract defined in ARXML, the application can have one or all three of the message types. Since the data can potentially flow over a network, the data is serialized/deserialized similar to other popular frameworks like FlatBuffers and ProtoBuf.  With multiple producers and consumers of data across a distributed system, time synchronization and Quality of Service (QOS) gain importance.

Time Synchronization and Network Management

Time synchronization plays a central role in the development of ADAS, or autonomous vehicles. For example, in sensor fusion where LiDAR cloud point data needs to be superimposed on a camera image, both the sensor data have to be synchronized to capture the data at the same instance. Moreover, if a system needs to have real-time constraints, it also needs to have the correct notion of time to ensure both logical and temporal correctness of the output. The adaptive AUTOSAR borrows upon the Synchronized Time-Base Manager specification from the classic variant with C++ Chrono-like support.

Overall, for correct behavior, there needs to be support in both hardware and software. A common protocol used in such scenarios is called PTP or Precision Time Protocol, capable of providing microsecond-level synchronization. PTP plays an important role in QOS. Recall that our distributed system has remote ECUs and the communication channel in modern AV or ADAS systems is based on an Ethernet bus with switches. 

Time Sensitive Networks (TSN) is a group of 802.1 IEEE standards. The standards allow for setting the QOS policy for data packets traveling over Ethernet. In the AV scenario, TSN with the help of PTP let us schedule the transmissions of data packets in a deterministic fashion by controlling the traffic flow and congestion.

Others

Persistency refers to AUTOSAR's ability to save data to a non-volatile memory. It offers a library for clients to save their data. It has the ability to save a key-value type of data or a FILE type of data similar to std::iostream.

The logging and diagnostic sub-module uses the communication manager APIs. External clients can interact with the sub-modules to request logs or implement special applications to gain insights into the running systems. AUTOSAR supports cryptography APIs to generate secure keys using algorithms such as RSA, AES, etc. The Cryptography Module may work alongside a Hardware Security Module to support security-sensitive operations. 

Conclusion

The classic AUTOSAR standard has been widely adopted in the development of automotives. The Adaptive standard has expanded to incorporate compute-intensive systems which allows for AV and ADAS operations. Apart from AUTOSAR Adaptive, ROS2 and DDS (Data Distribution Service) are some other middleware frameworks widely being used. There are efforts to get safety certification for ROS2 and DDS to be able to be used in autonomous vehicles. Despite which framework is used, AV has the potential to be a game-changer in the transportation industry and as problems are solved, new problems and use cases emerge which will keep this space as an exciting area to follow.