Disengagement Analysis and Field Tests of a Prototypical Open-Source Level 4 Autonomous Driving System

Abstract

Proprietary Autonomous Driving Systems are typically evaluated through disengagements, unplanned manual interventions to alter vehicle behavior, as annually reported by the California Department of Motor Vehicles. However, the real-world capabilities of prototypical open-source Level 4 vehicles over substantial distances remain largely unexplored. This study evaluates a research vehicle running an Autoware-based software stack across 236 km of mixed traffic. By classifying 30 disengagements across 26 rides with a novel five-level criticality framework, we observed a spatial disengagement rate of 0.127 1/km. Interventions predominantly occurred at lower speeds near static objects and traffic lights. Perception and Planning failures accounted for 40% and 26.7% of disengagements, respectively, largely due to object-tracking losses and operational deadlocks caused by parked vehicles. Frequent, unnecessary interventions highlighted a lack of trust on the part of the safety driver. These results show that while open-source software enables extensive operations, disengagement analysis is vital for uncovering robustness issues missed by standard metrics.

Figures (8)

• Figure 1: Overview of the real-world evaluation of an open-source Level 4 ads. (a) A prototypical research vehicle running the Autoware Universe stack was deployed across 236km of mixed-traffic environments. (b) Quantitative analysis of 30 disengagements revealed a spatial rate of 0.127km, with perception (40%) and planning (26.7%) limitations as the primary bottlenecks. (c) In-depth analysis exposes critical real-world situations, such as the perception system failing to detect a bus, demonstrating that rare anomalies can aggregate into system-level failures.
• Figure 2: Overview of the 9.9km suburban test route used for field evaluations. The route encompasses inner-city streets (blue), rural roads (green), and a highway segment (orange). Icons indicate the locations of traffic lights and roundabouts. Colored star markers denote the spatial distribution of the 30 recorded disengagement events, categorized by criticality as detailed in \ref{['tab:criticality_levels']}. Map data: © OpenStreetMap contributors © CARTO.
• Figure 3: Vehicle dynamics across 26 autonomous rides. The panels display velocity $v$, longitudinal acceleration $a_x$, and lateral acceleration $a_y$ as a function of driven distance $s$. Individual rides are shown in gray, with the mean profile represented by the solid blue line. Background shading indicates the operational environment: city (blue), highway (orange), and rural (green).
• Figure 4: Histogram plots and Kernel Density Estimates (KDE) Scott2015 for the velocities of all timestamps for all rides (blue) and the velocities at the moment of the disengagements (red).
• Figure 5: Temporal disengagement rate $\lambda_{T_i}$ for the hours of a day. The bars show the disengagement rate for a specific hour, while the annotations show the number of disengagements and the total number of minutes driven autonomously within these hours, across all rides over multiple days.