Learning to Drive from a World Model

TL;DR

This work tackles the challenge of end-to-end autonomous driving by training driving policies directly from real human data in on-policy simulation, thereby reducing reliance on hand-crafted perception and rules. It introduces two data-driven simulators—the traditional reprojection-based view synthesis and a learned Future Anchored World Model—both capable of grounding policy decisions in realistic driving scenarios and enabling ground-truth action supervision. The authors demonstrate that policies trained with these simulators can learn normal driving behavior and be deployed as ADAS in real vehicles (e.g., openpilot), though on-policy World Model training shows robust real-world performance and careful handling of sim-to-real gaps. The findings highlight the potential of data-driven, on-policy simulation to scale end-to-end driving while outlining limitations of reprojection and promising scalability of world-model approaches for future work in longitudinal policies and broader ADAS applications.

Abstract

Most self-driving systems rely on hand-coded perception outputs and engineered driving rules. Learning directly from human driving data with an end-to-end method can allow for a training architecture that is simpler and scales well with compute and data. In this work, we propose an end-to-end training architecture that uses real driving data to train a driving policy in an on-policy simulator. We show two different methods of simulation, one with reprojective simulation and one with a learned world model. We show that both methods can be used to train a policy that learns driving behavior without any hand-coded driving rules. We evaluate the performance of these policies in a closed-loop simulation and when deployed in a real-world advanced driver-assistance system.

Figures (11)

• Figure 1: One step of the World Model Simulation rollout. Gray filled shapes are inputs to the World Model. Black filled shapes are inputs to both the Policy Model and the World Model (note that the Policy Model can be the World Model itself). Circles are actions (positions and orientations) and rectangles are observations (images).
• Figure 2: Left: Top: Image at $T$. Bottom: Depth map at $T$. Right: Reprojected images at $T$ using 4 different translation vectors.
• Figure 3: Left: Image at $T$, Right: Reprojected Images at $T$. Notice the lighting artifacts in the reprojected images.
• Figure 4: Five examples of World Model simulation. Blue bordered frames are the last frames of the past context, red bordered frames are the first frames of the future anchoring, and green bordered frames are simulated frames. Notice how the simulated frames comply with the future anchoring by executing lanes changes, or turning the traffic light to green.
• Figure 5: Left: LPIPS for different DiT model sizes, trained on 400k segments. Right: LPIPS for different dataset sizes, for a DiT of 500M parameters. Both are from the action teacher-forced sequential rollout setting.