StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models

TL;DR

StreetCrafter introduces a LiDAR-conditioned controllable video diffusion model for street-view synthesis, enabling precise viewpoint control and pixel-level scene edits without per-scene optimization. It couples a diffusion backbone with LiDAR-derived per-frame conditions and distills the learned prior into a dynamic 3D Gaussian Splatting representation for real-time rendering, achieving strong extrapolation performance on Waymo Open and PandaSet. The method supports editing operations by adjusting LiDAR inputs and demonstrates robustness to dynamic elements, while acknowledging data-collection costs and current inference speed as areas for improvement. Overall, StreetCrafter delivers photorealistic, controllable view synthesis with practical editing capabilities for autonomous-driving simulation and analysis.

Abstract

This paper aims to tackle the problem of photorealistic view synthesis from vehicle sensor data. Recent advancements in neural scene representation have achieved notable success in rendering high-quality autonomous driving scenes, but the performance significantly degrades as the viewpoint deviates from the training trajectory. To mitigate this problem, we introduce StreetCrafter, a novel controllable video diffusion model that utilizes LiDAR point cloud renderings as pixel-level conditions, which fully exploits the generative prior for novel view synthesis, while preserving precise camera control. Moreover, the utilization of pixel-level LiDAR conditions allows us to make accurate pixel-level edits to target scenes. In addition, the generative prior of StreetCrafter can be effectively incorporated into dynamic scene representations to achieve real-time rendering. Experiments on Waymo Open Dataset and PandaSet demonstrate that our model enables flexible control over viewpoint changes, enlarging the view synthesis regions for satisfying rendering, which outperforms existing methods.

Figures (17)

• Figure 1: StreetCrafter is a novel controllable video diffusion model, which enables precise pose controllability for novel view synthesis in street scenes, using calibrated images and LiDAR as input. StreetCrafter can also serve as data prior to improve the scene reconstruction quality and support scene editing operations without per-scene optimization, such as object removal and replacement.
• Figure 2: Overview of StreetCrafter. (a) We process the LiDAR using calibrated images and object tracklets to obtain a colorized point cloud, which can be rendered to image space as pixel-level conditions. (b) Given observed images and reference image embedding $\mathbf{c}_\text{ref}$, we optimize the video diffusion model conditioned on the LiDAR renderings to perform controllable video generation. (c) Starting from the rendered images and LiDAR conditions under novel trajectory, we use the pretrained controllable video diffusion model to guide the optimization of the dynamic 3DGS representation by generating novel views as extra supervision signals.
• Figure 3: Qualitative comparisons on the Waymo Sun_2020_CVPR dataset. The camera is laterally shifted for 3 meters to left or right. Input view refers to the closest training camera. Ours-G denotes the dynamic 3DGS distilled from StreetCrafter and Ours-V denotes StreetCrafter.
• Figure 4: Qualitative comparisons on the PandaSet xiao2021pandaset dataset. The camera is laterally shifted for 3 meters to left or right.
• Figure 5: Visual ablation results on the design choice of StreetCrafter. The results indicate that our LiDAR condition can provide more accurate control for street view synthesis even when the viewpoint deviates greatly from the reference image.