DriveExplorer: Images-Only Decoupled 4D Reconstruction with Progressive Restoration for Driving View Extrapolation

TL;DR

This work addresses the challenge of extrapolating driving scene views without relying on expensive priors such as LiDAR or lane annotations. It introduces an images-only pipeline that separately estimates static and dynamic geometry, models dynamics with a 4D Gaussian framework, and trains a video diffusion model conditioned on pseudo-images and dynamic masks with progressively shifted viewpoints. The approach yields higher-quality lane-shifted renderings than priors-based baselines, validated on EUVS with ablations confirming the value of dense initial points, dynamic deformation, and multi-conditional conditioning. By removing the need for heavy sensor data and labels, it enables practical closed-loop driving simulation and planning with reduced data collection requirements.

Abstract

This paper presents an effective solution for view extrapolation in autonomous driving scenarios. Recent approaches focus on generating shifted novel view images from given viewpoints using diffusion models. However, these methods heavily rely on priors such as LiDAR point clouds, 3D bounding boxes, and lane annotations, which demand expensive sensors or labor-intensive labeling, limiting applicability in real-world deployment. In this work, with only images and optional camera poses, we first estimate a global static point cloud and per-frame dynamic point clouds, fusing them into a unified representation. We then employ a deformable 4D Gaussian framework to reconstruct the scene. The initially trained 4D Gaussian model renders degraded and pseudo-images to train a video diffusion model. Subsequently, progressively shifted Gaussian renderings are iteratively refined by the diffusion model,and the enhanced results are incorporated back as training data for 4DGS. This process continues until extrapolation reaches the target viewpoints. Compared with baselines, our method produces higher-quality images at novel extrapolated viewpoints.

Figures (3)

• Figure 1: (a) Data processing pipeline. We decouple static and dynamic point clouds directly from images, and project the fused global point cloud onto both the original viewpoints and shifted test viewpoints. (b) Gaussian training pipeline. We first train the dynamic Gaussians to render masks at the original viewpoints, ensuring the ability to generate dynamic masks from any viewpoint thereafter. We then deform the dynamic Gaussians using a 4D grid and couple them with the static Gaussians.
• Figure 2: Training pipeline of the video diffusion model. The pseudo-images and Gaussian dynamic mask obtained from Fig. \ref{['data-process']} are used as conditional inputs, concatenated with the latent features, and then passed to the denoising U-Net.
• Figure 3: Qualitative results on EUVS dataset.