Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Survey

TL;DR

This survey presents a comprehensive overview of learning-based 3D reconstruction for autonomous driving, tracing the evolution from neural radiance fields to efficient 3D Gaussians and 4D dynamic representations. It systematically organizes methods by driving-scene components, geometry and temporal modeling, and downstream applications in data collection, mapping, simulation, and scene understanding. The authors highlight key challenges, notably limited on-board validation and safety verification, and offer directions toward safer, real-time, and scalable deployment, including simulator-in-loop frameworks and physics-informed rendering. Overall, the work clarifies how advanced 3D representations can serve as a unified asset for perception, planning, and simulation in autonomous driving.

Abstract

Learning-based 3D reconstruction has emerged as a transformative technique in autonomous driving, enabling precise modeling of environments through advanced neural representations. It has inspired pioneering solutions for vital tasks in autonomous driving, such as dense mapping and closed-loop simulation, as well as comprehensive scene feature for driving scene understanding and reasoning. Given the rapid growth in related research, this survey provides a comprehensive review of both technical evolutions and practical applications in autonomous driving. We begin with an introduction to the preliminaries of learning-based 3D reconstruction to provide a solid technical background foundation, then progress to a rigorous, multi-dimensional examination of cutting-edge methodologies, systematically organized according to the distinctive technical requirements and fundamental challenges of autonomous driving. Through analyzing and summarizing development trends and cutting-edge research, we identify existing technical challenges, along with insufficient disclosure of on-board validation and safety verification details in the current literature, and ultimately suggest potential directions to guide future studies.

Figures (10)

• Figure 1: Survey outlines. We start from preliminaries and essentials of 3D reconstruction, then elaborate on the technical evolutions of traffic elements and dynamic driving scene reconstruction with tailored taxonomy, and diverse applications within autonomous driving. Finally, we summarize and delineate challenges and future directions. Click the section title or icon to jump to the corresponding section.
• Figure 2: illustrations of different representations. Top: Implicit Representations; Bottom: Explicit Representations.
• Figure 3: Rendering Pipeline of NeRF nerf and 3D Gaussian Splatting 3dgs
• Figure 4: Distance-based Segmentation of StreetSurf guo2023streetsurf
• Figure 5: 3D reconstruction of traffic agents.