Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving

TL;DR

This work tackles causal confusion in end-to-end autonomous driving by recognizing that imitation-learning supervision from noisy expert trajectories impedes true causal understanding. It proposes Perception-Guided Self-Supervision (PGS), which treats perception outputs as primary supervisory signals and introduces three mechanisms—Multi-Modal Trajectory Planning Self-Supervision (MTPS), Spatial Trajectory Planning Self-Supervision (STPS), and Negative Trajectory Planning Self-Supervision (NTPS)—to align planning with road topology and dynamic interactions. The training objective fuses baseline perception/motion losses with these self-supervision terms via $L'_{ ext{total}} = L_{ ext{total}} + w_{ ext{MTPS}} L_{ ext{MTPS}} + w_{ ext{STPS}} L_{ ext{STPS}} + w_{ ext{NTPS}} L_{ ext{NTPS}}$, enabling stronger causal modeling. On the Bench2Drive benchmark, PGS achieves a Driving Score of $78.08$ and a mean Success Rate of $48.64 ext{%}$, outperforming state-of-the-art methods with greater architectural simplicity. The results suggest that perception-aligned self-supervision can substantially improve real-world robustness and generalization in end-to-end autonomous driving.

Abstract

End-to-end autonomous driving systems, predominantly trained through imitation learning, have demonstrated considerable effectiveness in leveraging large-scale expert driving data. Despite their success in open-loop evaluations, these systems often exhibit significant performance degradation in closed-loop scenarios due to causal confusion. This confusion is fundamentally exacerbated by the overreliance of the imitation learning paradigm on expert trajectories, which often contain unattributable noise and interfere with the modeling of causal relationships between environmental contexts and appropriate driving actions. To address this fundamental limitation, we propose Perception-Guided Self-Supervision (PGS) - a simple yet effective training paradigm that leverages perception outputs as the primary supervisory signals, explicitly modeling causal relationships in decision-making. The proposed framework aligns both the inputs and outputs of the decision-making module with perception results, such as lane centerlines and the predicted motions of surrounding agents, by introducing positive and negative self-supervision for the ego trajectory. This alignment is specifically designed to mitigate causal confusion arising from the inherent noise in expert trajectories. Equipped with perception-driven supervision, our method, built on a standard end-to-end architecture, achieves a Driving Score of 78.08 and a mean success rate of 48.64% on the challenging closed-loop Bench2Drive benchmark, significantly outperforming existing state-of-the-art methods, including those employing more complex network architectures and inference pipelines. These results underscore the effectiveness and robustness of the proposed PGS framework and point to a promising direction for addressing causal confusion and enhancing real-world generalization in autonomous driving.

Figures (10)

• Figure 1: Overall Model Architecture. (1) Perception network provides map elements and future motions of dynamic agents. (2) The dashed box presents the proposed PGS, which generates three self-supervised signals from perception outputs to enhance causal reasoning in ego planning.
• Figure 2: Environment-Aware Lane Command and Road Topology Guided Planning.
• Figure 3: Negative Trajectory Planning Self-Supervision (NTPS) for Safety-Constrained Ego.
• Figure 4: Visualization of lane centerline perception under diverse road conditions
• Figure 5: Visualization of Target Lane Selection. Light pink points indicate the perceived lane centerlines, while dark red points represent the selected relevant lane set. The final target lane is also highlighted in dark red, with its name labeled in orange.