WorldWarp: Propagating 3D Geometry with Asynchronous Video Diffusion

TL;DR

WorldWarp tackles long-range, camera-conditioned novel-view video synthesis by integrating an online 3DGS geometric cache with a non-causal Spatio-Temporal Diffusion (ST-Diff) refiner. It abandons strict causal AR generation in favor of forward-warped priors and a fill-and-revise diffusion process that applies spatially-varying noise to preserve geometry while hallucinating new content in occluded regions. An online 3DGS-based cache maintains geometry fidelity across chunks, mitigating irreversible error propagation during long trajectories. Empirical results on RealEstate10K and DL3DV show state-of-the-art geometric consistency and visual fidelity, with ablations confirming the necessity of the 3D cache and the spatial-temporal noise schedule. This approach enables robust, geometry-guided long-range video generation conditioned on camera movement and prompts.

Abstract

Generating long-range, geometrically consistent video presents a fundamental dilemma: while consistency demands strict adherence to 3D geometry in pixel space, state-of-the-art generative models operate most effectively in a camera-conditioned latent space. This disconnect causes current methods to struggle with occluded areas and complex camera trajectories. To bridge this gap, we propose WorldWarp, a framework that couples a 3D structural anchor with a 2D generative refiner. To establish geometric grounding, WorldWarp maintains an online 3D geometric cache built via Gaussian Splatting (3DGS). By explicitly warping historical content into novel views, this cache acts as a structural scaffold, ensuring each new frame respects prior geometry. However, static warping inevitably leaves holes and artifacts due to occlusions. We address this using a Spatio-Temporal Diffusion (ST-Diff) model designed for a "fill-and-revise" objective. Our key innovation is a spatio-temporal varying noise schedule: blank regions receive full noise to trigger generation, while warped regions receive partial noise to enable refinement. By dynamically updating the 3D cache at every step, WorldWarp maintains consistency across video chunks. Consequently, it achieves state-of-the-art fidelity by ensuring that 3D logic guides structure while diffusion logic perfects texture. Project page: \href{https://hyokong.github.io/worldwarp-page/}{https://hyokong.github.io/worldwarp-page/}.

Figures (9)

• Figure 1: WorldWarp: Long-range novel view synthesis from a single image. Given only a single starting image (left) and a specified camera trajectory, our method generates a long and coherent video sequence. The core of our approach is to generate the video chunk-by-chunk, where each new chunk is conditioned on forward-warped "hints" from the previous one. A novel diffusion model then generates the next chunk by correcting these hints and filling in occlusions using a spatio-temporal varying noise schedule. The high geometric consistency of our 200-frame generated sequence is demonstrated by its successful reconstruction into a high-fidelity 3D Gaussian Splatting (3DGS) 3dgs model (right). This highlights our model's robust understanding of 3D geometry and its capability to maintain long-term consistency.
• Figure 2: Training pipeline of our ST-Diff model.1) Spatially temporally-varying noisy latent: The process begins by rendering a warped image and a validity mask from an RGB point cloud (images are shown for illustration, as operations are in latent space). The warped image is encoded to get $\mathbf{z}_{s \to t}$, and the ground-truth image is encoded to get $\mathbf{z}_t$. A "clean composite" latent $\mathbf{z}_{c,t}$ is created by combining the valid warped regions from $\mathbf{z}_{s \to t}$ with the blank regions from $\mathbf{z}_t$, using the downsampled mask $\mathbf{M}_{\text{latent}}$. 2) Training ST-Diffusion: This composite latent sequence is noised according to our spatio-temporal schedule, resulting in a noisy latent sequence (visualized as a stack) where the noise level for each latent varies across different frames and spatial regions. The resulting noisy latents are fed into our model $\bm{G}_\theta$, which is trained to predict the target velocity (defined as $\mathbf{\epsilon}_t - \mathbf{z}_t$), forcing it to learn the flow from the noisy composite latent back towards the original ground-truth latent sequence $\mathcal{Z}$.
• Figure 3: The autoregressive inference pipeline of WorldWarp. At each iteration $k$, the available history (either the initial images or the previously generated $k-1$ chunk) is processed. First, TTT3R estimates camera poses and an initial 3D point cloud. This geometry is used to optimize a 3D Gaussian Splatting (3DGS) representation, which serves as a high-fidelity 3D cache. Concurrently, a VLM generates a descriptive text prompt, and novel camera poses are extrapolated for the next chunk. The optimized 3DGS renders forward-warped images at these new poses. These warped priors, along with the VLM prompt, are fed into our non-causal ST-Diff model ($G_\theta$) to denoise and generate the $k$-th chunk of novel views. The process then repeats, using the newly generated chunk as the history for the next iteration.
• Figure 4: Qualitative comparisons on the RealEstate10K re10k and DL3DV dl3dv datasets. We visualize videos generated by our method against those by GenWarp genwarp, CameraCtrl cameractrl, and VMem vmem. Our WorldWarp generalizes to diverse camera motion, showcasing the spatial and temporal consistency.
• Figure 5: Illustration of the ST-Diff's generating process. We illustrate the GT images, the warped images which serve as the condition for ST-Diff, the corresponding validity mask, and our final generated frames. The comparisons show that our ST-Diff successfully fills in the blank areas (initialized from a full noise level) while simultaneously revising distortions and enhancing details in the non-blank regions (initialized from a partial noise level) during the diffusion process.