ViewMorpher3D: A 3D-aware Diffusion Framework for Multi-Camera Novel View Synthesis in Autonomous Driving

TL;DR

The paper tackles artifacts in novel-view synthesis for autonomous driving caused by sparse or extrapolated observations. It introduces ViewMorpher3D, a diffusion-based multi-view enhancer that jointly processes a variable number of reference and target views, conditioned on 3D priors (C-maps) and Plücker ray embeddings alongside camera poses. Training uses latent-space supervision for all targets with selective pixel-space supervision and includes LoRA-based fine-tuning of the VAE decoder to boost fidelity. Across EUVS, Para-Lane, nuScenes, and DL3DV, ViewMorpher3D achieves superior perceptual quality and geometric coherence under view extrapolation, enabling more reliable closed-loop simulation and planning for autonomous driving.

Abstract

Autonomous driving systems rely heavily on multi-view images to ensure accurate perception and robust decision-making. To effectively develop and evaluate perception stacks and planning algorithms, realistic closed-loop simulators are indispensable. While 3D reconstruction techniques such as Gaussian Splatting offer promising avenues for simulator construction, the rendered novel views often exhibit artifacts, particularly in extrapolated perspectives or when available observations are sparse. We introduce ViewMorpher3D, a multi-view image enhancement framework based on image diffusion models, designed to elevate photorealism and multi-view coherence in driving scenes. Unlike single-view approaches, ViewMorpher3D jointly processes a set of rendered views conditioned on camera poses, 3D geometric priors, and temporally adjacent or spatially overlapping reference views. This enables the model to infer missing details, suppress rendering artifacts, and enforce cross-view consistency. Our framework accommodates variable numbers of cameras and flexible reference/target view configurations, making it adaptable to diverse sensor setups. Experiments on real-world driving datasets demonstrate substantial improvements in image quality metrics, effectively reducing artifacts while preserving geometric fidelity.

Figures (10)

• Figure 1: ViewMorpher3D improves rendered novel views via multi-view diffusion, conditioned on camera images, poses, and the scene's 3D structure.
• Figure 2: Overview illustration of ViewMorpher3D. The rendered novel-view images are enhanced via a multi-view diffusion model, conditioned on reference views, camera poses and 3D priors.
• Figure 3: Novel-view images of EVUS (left 4 col.) and Para-Lane (right 2 col.) datasets; The qualitative comparison of enhanced 3DGS rendered images from DiFix3D and ViewMorpher3D. The ViewMorpher3D results show a higher fidelity to the scene visual contents.
• Figure 4: NVS image quality on nuScenes Results of overall performance on extrapolated trajectories (up to 5 seconds).
• Figure 5: NVS image quality under various signal-to-noise ration on nuScenes; The metrics for the extrapolated driving sequences (up to 5 seconds) are shown.