AutoScape: Geometry-Consistent Long-Horizon Scene Generation

TL;DR

AutoScape tackles long-horizon 3D‑consistent driving scene generation by a hierarchical two‑stage approach: first, a geometry‑aware RGB‑D diffusion model generates sparse, highly consistent keyframes conditioned on existing scene geometry; then a video diffusion model interpolates between these anchors to produce dense, coherent frames. The RGB‑D diffusion model adopts joint RGB‑D encoding, explicit geometry conditioning on rendered point clouds, and warp‑consistent guidance to maintain 3D alignment across steps. Two‑stage training (RGB‑D pretraining and rendering‑conditioned fine‑tuning) plus data filtering yield strong geometry priors, while the interpolation stage leverages a point‑cloud conditioned model to ensure global geometric coherence. They demonstrate substantial improvements over state‑of‑the‑art methods on long sequences (up to 20 seconds) in FID and FVD, with good generalization to new datasets and strong controllability via HD maps, object boxes, and text prompts. The work highlights a practical path toward robust, photo‑realistic, long‑horizon driving simulations for verification and robotics applications.

Abstract

This paper proposes AutoScape, a long-horizon driving scene generation framework. At its core is a novel RGB-D diffusion model that iteratively generates sparse, geometrically consistent keyframes, serving as reliable anchors for the scene's appearance and geometry. To maintain long-range geometric consistency, the model 1) jointly handles image and depth in a shared latent space, 2) explicitly conditions on the existing scene geometry (i.e., rendered point clouds) from previously generated keyframes, and 3) steers the sampling process with a warp-consistent guidance. Given high-quality RGB-D keyframes, a video diffusion model then interpolates between them to produce dense and coherent video frames. AutoScape generates realistic and geometrically consistent driving videos of over 20 seconds, improving the long-horizon FID and FVD scores over the prior state-of-the-art by 48.6\% and 43.0\%, respectively.

Figures (9)

• Figure 1: AutoScape generates long-horizon and 3D-consistent driving scenes from a single input image, producing high-quality videos over a temporal span of 20 seconds. Click the images in the right column within Adobe Reader to play three example videos.
• Figure 2: Pipeline of the AutoScape. The vehicle trajectory defines the location of keyframes and interpolation frames, spanning a long-horizon 3D space. The Keyframes Generation stage iteratively generates geometrically consistent keyframes with an RGB-D diffusion model as global scene anchors. The Interpolation stage then produces dense frames with a video diffusion model. The keyframe viewpoints are indicated by and the interpolation viewpoints are marked by , best viewed in color.
• Figure 3: (Left) Input processing of our RGB-D diffusion model. (Right) The warp-consistent guidance for steering the sampling process toward better geometric consistency.
• Figure 4: Qualitative comparison to show the effectiveness of our warp consistent guidance (WCG) strategy for better consistency.
• Figure 5: Qualitative comparisons. WonderJourney$^\dag$ represents WonderJourney adapted for driving scenes. We sample frames from different time splits to demonstrate both short and long-term performance. AutoScape maintains 3D consistency over significant view changes and extends toward a long horizon. Vista generates the video from the start of the trajectory to the end, while others generate in the reverse order.