Driving in Corner Case: A Real-World Adversarial Closed-Loop Evaluation Platform for End-to-End Autonomous Driving

TL;DR

The paper addresses the challenge of evaluating end-to-end autonomous driving under safety-critical corner cases by engineering a real-world adversarial closed-loop platform. It couples a flow-matching-based Real-World Image Generator with an adversarial traffic policy to create realistic, controllable scenes and interactive evaluation against E2E models. The authors demonstrate that adversarial traffic significantly degrades performance metrics for models like UniAD and VAD, highlighting weaknesses under corner cases and the platform's ability to reveal safety issues. The work provides a practical tool for stress-testing end-to-end driving systems and advancing safety and robustness for real-world deployment.

Abstract

Safety-critical corner cases, difficult to collect in the real world, are crucial for evaluating end-to-end autonomous driving. Adversarial interaction is an effective method to generate such safety-critical corner cases. While existing adversarial evaluation methods are built for models operating in simplified simulation environments, adversarial evaluation for real-world end-to-end autonomous driving has been little explored. To address this challenge, we propose a closed-loop evaluation platform for end-to-end autonomous driving, which can generate adversarial interactions in real-world scenes. In our platform, the real-world image generator cooperates with an adversarial traffic policy to evaluate various end-to-end models trained on real-world data. The generator, based on flow matching, efficiently and stably generates real-world images according to the traffic environment information. The efficient adversarial surrounding vehicle policy is designed to model challenging interactions and create corner cases that current autonomous driving systems struggle to handle. Experimental results demonstrate that the platform can generate realistic driving images efficiently. Through evaluating the end-to-end models such as UniAD and VAD, we demonstrate that based on the adversarial policy, our platform evaluates the performance degradation of the tested model in corner cases. This result indicates that this platform can effectively detect the model's potential issues, which will facilitate the safety and robustness of end-to-end autonomous driving.

Figures (5)

• Figure 1: Overview of the real-world adversarial closed-loop evaluation platform. The platform integrates three key modules: Adversarial Traffic Flow, Real-World Image Generator, and E2E Tested Model. The Adversarial Traffic Flow generates surrounding vehicles that interact adversarially with the ego, providing traffic information to the Real-World Image Generator. The generator efficiently generates real-world images based on traffic information through flow matching. The generated images are passed as input to the E2E Tested Model, and the model's output is fed back to the Adversarial Traffic Flow, completing the closed-loop simulation.
• Figure 2: Adversarial surrounding vehicle generation method. The method consists of two episodes. The first episode replays a steady traffic flow, and the trajectory of the tested model is recorded. Based on the recorded data, an adversarial and physically plausible trajectory of the surrounding vehicle is selected, and then this trajectory is applied in the second episode.
• Figure 3: Overview of the Real-World Image Generator. The backbone network of flow matching is a UNet, which leverages diffusion priors through linear transformation. Information projected into the camera view is injected via ControlNet, while other conditional information is incorporated through attention mechanisms.
• Figure 4: Comparison of generated image quality. We generate three sets of example images: a, b, and c. In sub-figures a and b, the elements within the red boxes clearly show that our generator generates higher-quality images. In sub-figure c, it can be observed that both the front and back views are consistently rainy, while the baseline shows noticeable differences.
• Figure 5: A typical case in adversarial closed-loop evaluating. The top and bottom sections show the performance of UniAD and VAD, and we capture three key frames from the interaction. In each cell, the left side displays the ground truth traffic flow extracted from MetaDrive. The center shows the generated image from the Real-World Image Generator. The right side displays the output of the tested end-to-end model.