LaVR: Scene Latent Conditioned Generative Video Trajectory Re-Rendering using Large 4D Reconstruction Models

TL;DR

LaVR introduces latent-space conditioning for video trajectory re-rendering by leveraging the latent state of a large 4D reconstruction model (via CUT3R) to provide implicit geometric priors without relying on depth estimates. A lightweight CUT3R adapter integrates these latents with a DiT-based diffusion backbone, and a conditional flow-matching objective guides synthesis on synthetic MultiCamVideo data. The approach delivers superior geometric fidelity and visual quality across dynamic scenes, outperforming both geometry-conditioned point-cloud methods and unconditioned baselines, while remaining parameter-efficient (~1.3B) compared to larger models. Limitations include challenges with dynamic transparent objects, but the method sets a new benchmark for stable parallax and scene-consistent novel-view video from monocular input.

Abstract

Given a monocular video, the goal of video re-rendering is to generate views of the scene from a novel camera trajectory. Existing methods face two distinct challenges. Geometrically unconditioned models lack spatial awareness, leading to drift and deformation under viewpoint changes. On the other hand, geometrically-conditioned models depend on estimated depth and explicit reconstruction, making them susceptible to depth inaccuracies and calibration errors. We propose to address these challenges by using the implicit geometric knowledge embedded in the latent space of a large 4D reconstruction model to condition the video generation process. These latents capture scene structure in a continuous space without explicit reconstruction. Therefore, they provide a flexible representation that allows the pretrained diffusion prior to regularize errors more effectively. By jointly conditioning on these latents and source camera poses, we demonstrate that our model achieves state-of-the-art results on the video re-rendering task. Project webpage is https://lavr-4d-scene-rerender.github.io/

Figures (9)

• Figure 1: Our method addresses the problem of rendering geometrically consistent novel trajectories from a monocular source video. We propose to utilize the geometric knowledge of a pre-trained large reconstruction model (LRM) by conditioning the trajectory generation process on the latent state of a 4D LRM. Compared to prior methods which are conditioned on error-prone point cloud re-renderings of the source video, our method achieves state-of-the-art visual quality while maintaining a high level of geometric fidelity to the original scene.
• Figure 2: Architecture comparison.(a) Unconditioned methods for novel trajectory generation achieve high visual quality but lack geometric awareness, leading to inconsistencies. (b) Conditioning on 4D point cloud renders provides consistency, but reduces quality as the depth estimation and point cloud generation stages are sensitive to errors. (c) Our proposed architecture utilizes the implicit geometric knowledge of a pre-trained large 4D reconstruction model (LRM) to achieve both high quality and consistency.
• Figure 3: Pipeline overview. Given a monocular source video, our method generates a novel video of the same scene at a target camera trajectory using a video diffusion model. To ensure geometric consistency, we condition the model on latents from CUT3R cut3r, a pre-trained 4D reconstruction model. We use four signals from the source video: the standard video VAE latents, CUT3R's 4D latents, source camera poses, and an encoded text description of the scene. A novel adapter architecture aligns the CUT3R and VAE latents, and allows these to be fed to the model in a computationally feasible manner. The source camera poses come from CUT3R, and are added to the DiT's intermediate activations after passing through a small MLP-based adapter. Another MLP processes the target poses at which the novel video is rendered. Note that only the projection and self-attention layers of the DiT are trainable, other parameters are frozen.
• Figure 4: Proposed CUT3R Adapter. Our lightweight adapter compresses CUT3R's per-frame latent tokens into geometry-aware features that align with the representation used by the diffusion model. The shape of features at each stage is shown in brackets.
• Figure 5: Evaluation on the VBench huang2023vbenchzheng2025vbench2 suite of metrics. We highlight relative differences by normalizing each metric over all baselines. Our method shows all-around high performance, achieving the best results for multi-view, subject, and background consistency.