DiffE2E: Rethinking End-to-End Driving with a Hybrid Action Diffusion and Supervised Policy

TL;DR

DiffE2E tackles multimodal end-to-end autonomous driving by marrying diffusion-based trajectory generation with explicit supervision in a Transformer framework. It introduces a hierarchical cross-modal fusion module and a hybrid diffusion–supervision decoder, enabling joint optimization of a multimodal trajectory distribution and controllable outputs through cross-attention with latent variables. The approach achieves state-of-the-art performance on CARLA and NAVSIM benchmarks, supported by a two-stage training regime and rigorous ablations that justify the hybrid design. This structured latent-space diffusion paradigm offers strong generalization and real-time capability, with potential extensions to broader embodied intelligence domains.

Abstract

End-to-end learning has emerged as a transformative paradigm in autonomous driving. However, the inherently multimodal nature of driving behaviors and the generalization challenges in long-tail scenarios remain critical obstacles to robust deployment. We propose DiffE2E, a diffusion-based end-to-end autonomous driving framework. This framework first performs multi-scale alignment of multi-sensor perception features through a hierarchical bidirectional cross-attention mechanism. It then introduces a novel class of hybrid diffusion-supervision decoders based on the Transformer architecture, and adopts a collaborative training paradigm that seamlessly integrates the strengths of both diffusion and supervised policy. DiffE2E models structured latent spaces, where diffusion captures the distribution of future trajectories and supervision enhances controllability and robustness. A global condition integration module enables deep fusion of perception features with high-level targets, significantly improving the quality of trajectory generation. Subsequently, a cross-attention mechanism facilitates efficient interaction between integrated features and hybrid latent variables, promoting the joint optimization of diffusion and supervision objectives for structured output generation, ultimately leading to more robust control. Experiments demonstrate that DiffE2E achieves state-of-the-art performance in both CARLA closed-loop evaluations and NAVSIM benchmarks. The proposed integrated diffusion-supervision policy offers a generalizable paradigm for hybrid action representation, with strong potential for extension to broader domains including embodied intelligence. More details and visualizations are available at \href{https://infinidrive.github.io/DiffE2E/}{project website}.

Figures (8)

• Figure 1: Comparison of end-to-end training paradigms. (a) Explicit Policy. Directly predicts trajectories through supervised learning after sensor input processing. (b) Explicit Policy with Diffusion Refinement. Uses diffusion models to replace traditional explicit policy trajectory output heads. (c) Diffusion Policy. Uses diffusion models to directly generate trajectories based on perception encoder features.
• Figure 2: Overall architecture of DiffE2E. The main architecture consists of a Transformer-based perception module and a Hybrid Diffusion and Supervision Decoder. The blue arrows ($\bm{\textcolor{customblue}{\rightarrow}}$) indicate the data flow exclusively used for the CARLA benchmark, while the black arrows ($\bm{\rightarrow}$) represent the data flow shared between both the CARLA and NAVSIM benchmarks.
• Figure 3: Visualization in CARLA Simulator. In both the LiDAR and scene visualizations, blue points represent the predicted trajectory, while red points in the LiDAR view denote the target waypoints.
• Figure 3: Ablation results of DiffE2E on CARLA Longest6 benchmark.
• Figure 4: Visualization in Navtest benchmark. Red trajectories denote the predicted paths of each method, while green trajectory corresponds to the ground truth.