Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative AI for Autonomous Driving: Frontiers and Opportunities

Yuping Wang, Shuo Xing, Cui Can, Renjie Li, Hongyuan Hua, Kexin Tian, Zhaobin Mo, Xiangbo Gao, Keshu Wu, Sulong Zhou, Hengxu You, Juntong Peng, Junge Zhang, Zehao Wang, Rui Song, Mingxuan Yan, Walter Zimmer, Xingcheng Zhou, Peiran Li, Zhaohan Lu, Chia-Ju Chen, Yue Huang, Ryan A. Rossi, Lichao Sun, Hongkai Yu, Zhiwen Fan, Frank Hao Yang, Yuhao Kang, Ross Greer, Chenxi Liu, Eun Hak Lee, Xuan Di, Xinyue Ye, Liu Ren, Alois Knoll, Xiaopeng Li, Shuiwang Ji, Masayoshi Tomizuka, Marco Pavone, Tianbao Yang, Jing Du, Ming-Hsuan Yang, Hua Wei, Ziran Wang, Yang Zhou, Jiachen Li, Zhengzhong Tu

TL;DR

Generative AI offers a unifying, data-efficient pathway to advance autonomous driving beyond conventional modular pipelines by enabling high-fidelity sensor synthesis, world modeling, and end-to-end decision-making. The survey maps foundational models (VAEs, GANs, diffusion, NeRF, SMPL, LLMs) to driving applications across perception, prediction, planning, and validation, and outlines real-world deployments via synthetic data, digital twins, and multimodal reasoning. It highlights opportunities in long-tail scenario generation, sim-to-real transfer, and V2X collaboration, while emphasizing safety, governance, transparency, and environmental considerations as prerequisites for trustworthy adoption. By consolidating technical progress with socio-technical implications, the work provides a roadmap for researchers, practitioners, and policymakers toward scalable, safe, and ethical GenAI-enabled autonomous mobility. The integration of GenAI across AD stacks promises faster development cycles, richer simulation environments, and more adaptable, human-centered transportation systems, albeit under careful oversight to manage risk, accountability, and equity.

Abstract

Generative Artificial Intelligence (GenAI) constitutes a transformative technological wave that reconfigures industries through its unparalleled capabilities for content creation, reasoning, planning, and multimodal understanding. This revolutionary force offers the most promising path yet toward solving one of engineering's grandest challenges: achieving reliable, fully autonomous driving, particularly the pursuit of Level 5 autonomy. This survey delivers a comprehensive and critical synthesis of the emerging role of GenAI across the autonomous driving stack. We begin by distilling the principles and trade-offs of modern generative modeling, encompassing VAEs, GANs, Diffusion Models, and Large Language Models (LLMs). We then map their frontier applications in image, LiDAR, trajectory, occupancy, video generation as well as LLM-guided reasoning and decision making. We categorize practical applications, such as synthetic data workflows, end-to-end driving strategies, high-fidelity digital twin systems, smart transportation networks, and cross-domain transfer to embodied AI. We identify key obstacles and possibilities such as comprehensive generalization across rare cases, evaluation and safety checks, budget-limited implementation, regulatory compliance, ethical concerns, and environmental effects, while proposing research plans across theoretical assurances, trust metrics, transport integration, and socio-technical influence. By unifying these threads, the survey provides a forward-looking reference for researchers, engineers, and policymakers navigating the convergence of generative AI and advanced autonomous mobility. An actively maintained repository of cited works is available at https://github.com/taco-group/GenAI4AD.

Generative AI for Autonomous Driving: Frontiers and Opportunities

TL;DR

Generative AI offers a unifying, data-efficient pathway to advance autonomous driving beyond conventional modular pipelines by enabling high-fidelity sensor synthesis, world modeling, and end-to-end decision-making. The survey maps foundational models (VAEs, GANs, diffusion, NeRF, SMPL, LLMs) to driving applications across perception, prediction, planning, and validation, and outlines real-world deployments via synthetic data, digital twins, and multimodal reasoning. It highlights opportunities in long-tail scenario generation, sim-to-real transfer, and V2X collaboration, while emphasizing safety, governance, transparency, and environmental considerations as prerequisites for trustworthy adoption. By consolidating technical progress with socio-technical implications, the work provides a roadmap for researchers, practitioners, and policymakers toward scalable, safe, and ethical GenAI-enabled autonomous mobility. The integration of GenAI across AD stacks promises faster development cycles, richer simulation environments, and more adaptable, human-centered transportation systems, albeit under careful oversight to manage risk, accountability, and equity.

Abstract

Generative Artificial Intelligence (GenAI) constitutes a transformative technological wave that reconfigures industries through its unparalleled capabilities for content creation, reasoning, planning, and multimodal understanding. This revolutionary force offers the most promising path yet toward solving one of engineering's grandest challenges: achieving reliable, fully autonomous driving, particularly the pursuit of Level 5 autonomy. This survey delivers a comprehensive and critical synthesis of the emerging role of GenAI across the autonomous driving stack. We begin by distilling the principles and trade-offs of modern generative modeling, encompassing VAEs, GANs, Diffusion Models, and Large Language Models (LLMs). We then map their frontier applications in image, LiDAR, trajectory, occupancy, video generation as well as LLM-guided reasoning and decision making. We categorize practical applications, such as synthetic data workflows, end-to-end driving strategies, high-fidelity digital twin systems, smart transportation networks, and cross-domain transfer to embodied AI. We identify key obstacles and possibilities such as comprehensive generalization across rare cases, evaluation and safety checks, budget-limited implementation, regulatory compliance, ethical concerns, and environmental effects, while proposing research plans across theoretical assurances, trust metrics, transport integration, and socio-technical influence. By unifying these threads, the survey provides a forward-looking reference for researchers, engineers, and policymakers navigating the convergence of generative AI and advanced autonomous mobility. An actively maintained repository of cited works is available at https://github.com/taco-group/GenAI4AD.
Paper Structure (122 sections, 35 equations, 25 figures, 13 tables)

This paper contains 122 sections, 35 equations, 25 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Surveys
  3. Additional Surveys on Autonomous Driving and Generative Models
  4. Datasets and Benchmarks for Autonomous Driving
  5. Real-World Single-Vehicle Perception Datasets
  6. Real-World Multi-Agent Prediction & Planning Datasets
  7. Synthetic and Simulation Datasets
  8. Language Annotated Datasets
  9. Fundamentals of Generative AI
  10. Variational Autoencoder (VAE) and Variants
  11. Vanilla VAE
  12. Conditional Variational Autoencoder (CVAE)
  13. Vector Quantized Variational Autoencoders (VQ-VAE)
  14. Generative Adversarial Network (GAN) and Variants
  15. Vanilla GAN
  16. ...and 107 more sections

Figures (25)

  • Figure 1: Historical overview of autonomous driving development, illustrating key technological periods (from early ADAS to the generative AI era) alongside timelines for major commercial L4 players (e.g., Waymo, Apollo), selected L4 startups, and established L2/3 ADAS companies.
  • Figure 2: Overview of an autonomous vehicle's perception system, illustrating sensor fusion and coverage. Top: Typical placements for key sensors including cameras, LiDAR, and various radar types. Bottom: Functional coverage zones achieved through sensor fusion, showing areas responsible for tasks like collision avoidance, traffic sign recognition, and surround environmental mapping
  • Figure 3: Illustration of popular but easily confused generation models. The black arrows denote training time program flow, whereas the purple arrows denote inference time program flow.
  • Figure 4: Emergence of various generative AI methods and the evolution of generative AI in autonomous driving.
  • Figure 5: Illustration of controllable generation where BEV layout serves as the condition. The image is from swerdlow2024street.
  • ...and 20 more figures