Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving

TL;DR

Drive-JEPA advances end-to-end autonomous driving by marrying V-JEPA-based self-supervised video pretraining with multimodal trajectory distillation from simulators. The method learns planning-oriented representations from large-scale driving videos and uses a proposal-centric planner enriched with simulator-generated pseudo-teachers to produce diverse, safe trajectories, complemented by a momentum-aware selection that improves temporal smoothness. Empirical results on NAVSIM and Bench2Drive show state-of-the-art performance in both perception-free and perception-based settings, with notable gains in planning reliability and comfort. The approach offers a scalable, data-efficient path to multimodal planning and safer end-to-end driving without heavy perception annotations.

Abstract

End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art.

Figures (6)

• Figure 1: Comparison between end-to-end planners on both perception-free and perception-based settings.
• Figure 2: Overview of the Drive-JEPA architecture. Driving Video Pretraining learns a ViT encoder from large-scale driving videos using the self-supervised V-JEPA objective. Given the pretrained features, Waypoint-anchored Proposal Generation efficiently produces multiple trajectory proposals, whose distribution is guided by Multimodal Trajectory Distillation. Finally, Momentum-aware Trajectory Selection picks the final trajectory by accounting for cross-frame comfort.
• Figure 3: Bird's eye view of proposals.
• Figure 4: Qualitative comparison of trajectories by different models in front-facing camera and bird's eye view on different driving scenarios. Trajectories are shown for: Human Trajectory, Drive-JEPA, iPad, Transfuser.
• Figure 5: Multimodal Trajectory Distillation improves PDM score.