Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Survey

TL;DR

This survey analyzes the synergy between diffusion-based video generation and world models in autonomous driving, framing world models as perception-prediction and RL-based architectures and examining leading works such as JEPA, Genie, and Sora. It details how diffusion models and latent representations enable coherent, controllable scene synthesis and forecasting, drawing connections to driving-relevant tasks like scene generation and motion prediction. The paper catalogs datasets and metrics, discusses current challenges—data and compute demands, privacy, and architectural innovation—and highlights future directions in multimodal integration, diversified scene generation, and unsupervised learning. Overall, it argues that integrating video generation with world models can substantially improve training, evaluation, and safety in autonomous driving, while outlining practical path forward and research gaps.

Abstract

World models and video generation are pivotal technologies in the domain of autonomous driving, each playing a critical role in enhancing the robustness and reliability of autonomous systems. World models, which simulate the dynamics of real-world environments, and video generation models, which produce realistic video sequences, are increasingly being integrated to improve situational awareness and decision-making capabilities in autonomous vehicles. This paper investigates the relationship between these two technologies, focusing on how their structural parallels, particularly in diffusion-based models, contribute to more accurate and coherent simulations of driving scenarios. We examine leading works such as JEPA, Genie, and Sora, which exemplify different approaches to world model design, thereby highlighting the lack of a universally accepted definition of world models. These diverse interpretations underscore the field's evolving understanding of how world models can be optimized for various autonomous driving tasks. Furthermore, this paper discusses the key evaluation metrics employed in this domain, such as Chamfer distance for 3D scene reconstruction and Fréchet Inception Distance (FID) for assessing the quality of generated video content. By analyzing the interplay between video generation and world models, this survey identifies critical challenges and future research directions, emphasizing the potential of these technologies to jointly advance the performance of autonomous driving systems. The findings presented in this paper aim to provide a comprehensive understanding of how the integration of video generation and world models can drive innovation in the development of safer and more reliable autonomous vehicles.

Figures (4)

• Figure 1: Video generation tasks can typically be divided into traditional video generation and diffusion-based video generation, the latter of which has seen widespread adoption in recent years. This paper categorizes world models in the autonomous driving domain into Perception-Prediction Structure and Reinforcement Learning Structure, with the former having a pipeline highly analogous to diffusion-based video generation tasks. This connection allows the paper to focus on the interplay between video generation and world models. Applications of world models in the autonomous driving field include Scene Generation, Action Planning, and Environment Perception.
• Figure 2: The interpretations and pipeline designs of world models presented in the works JEPA, Genie, Sora and ADriver-I indicate that the concept of world models lacks a universally accepted definition. These three works not only highlight the diversity of thought within the field but also represent distinct approaches to understanding and implementing world models, each reflecting different conceptualizations and priorities in model design.
• Figure 3: Timeline of World Models in Autonomous Driving. Based on the key architectures, we categorize the models from various studies into Perception-Prediction structures and Reinforcement Learning structures. This paper primarily focuses on the former, as its interplay with video generation models is more prominent.
• Figure 4: The unified architecture of video generation and world models in autonomous driving. is generative models or world models with temporal prediction capabilities. Both methodologies encompass two primary components: the perception and simulation module and the core prediction module. The perception and simulation module is primarily responsible for learning data distributions, extracting real-world data. Conversely, the prediction module focuses on learning the dynamic patterns of data within the compressed space, thereby simulating data changes.