A Comprehensive Evaluation of Four End-to-End AI Autopilots Using CCTest and the Carla Leaderboard

TL;DR

This work scrutinizes end-to-end AI autopilots for autonomous driving by applying Critical Configuration Testing (CCTest) within the Carla simulation landscape and comparing results to both open modular baselines and Leaderboard-based evaluations. CCTest generates minimal, potentially safe yet critical configurations to qualitatively assess safety policies, while the Carla Leaderboard provides broad quantitative scores across diverse real-world-like scenarios. Across InterFuser, MILE, Transfuser, and LMDrive, the study reveals widespread safety issues in critical contexts, including accidents and traffic-rule violations, and highlights differences from modular autopilots in behavior such as route changes and road deviations. The paper argues for a combined evaluation approach that integrates objective, quantitative Leaderboard metrics with qualitative CCTest analyses to more faithfully assess ADS safety and performance, and it discusses the limitations and complementary strengths of each method.

Abstract

End-to-end AI autopilots for autonomous driving systems have emerged as a promising alternative to traditional modular autopilots, offering the potential to reduce development costs and mitigate defects arising from module composition. However, they suffer from the well-known problems of AI systems such as non-determinism, non-explainability, and anomalies. This naturally raises the question of their evaluation and, in particular, their comparison with existing modular solutions. This work extends a study of the critical configuration testing (CCTest) approach that has been applied to four open modular autopilots. This approach differs from others in that it generates test cases ensuring safe control policies are possible for the tested autopilots. This enables an accurate assessment of the ability to drive safely in critical situations, as any incident observed in the simulation involves the failure of a tested autopilot. The contribution of this paper is twofold. Firstly, we apply the CCTest approach to four end-to-end open autopilots, InterFuser, MILE, Transfuser, and LMDrive, and compare their test results with those of the four modular open autopilots previously tested with the same approach implemented in the Carla simulation environment. This comparison identifies both differences and similarities in the failures of the two autopilot types in critical configurations. Secondly, we compare the evaluations of the four autopilots carried out in the Carla Leaderboard with the CCTest results. This comparison reveals significant discrepancies, reflecting differences in test case generation criteria and risk assessment methods. It underlines the need to work towards the development of objective assessment methods combining qualitative and quantitative criteria.

Figures (9)

• Figure 1: Priority-protected meeting of routes
• Figure 2: test case with traffic light
• Figure 3: Simulating one vehicle in the Carla Simulator
• Figure 4: The Carla Leaderboard test framework
• Figure 5: The architecture of the CCTest Framework