Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Camera-Controlled Video Generation of Static Scenes via Sparse Diffusion and 3D Rendering

Jieying Chen, Jeffrey Hu, Joan Lasenby, Ayush Tewari

TL;DR

SRENDER addresses the bottleneck of diffusion-based video generation by combining sparse, camera-conditioned keyframe diffusion with deterministic 3D Gaussian Splatting reconstruction to render dense video of static scenes. A transformer-based adaptive keyframe density predictor allocates computation based on trajectory complexity, and a two-stage diffusion process maintains coherence across sparse keyframes. Dense 3D reconstruction via AnySplat, followed by efficient 3DGS rendering and temporal chunking, yields substantial speedups (over 20–40×) while preserving or improving FID and FVD relative to strong diffusion baselines. The approach leverages explicit 3D reasoning to reduce computation dramatically, enabling real-time, controllable video synthesis for applications in embodied AI and VR/AR, with potential extension to dynamic scenes as 4D reconstruction advances mature.

Abstract

Modern video generative models based on diffusion models can produce very realistic clips, but they are computationally inefficient, often requiring minutes of GPU time for just a few seconds of video. This inefficiency poses a critical barrier to deploying generative video in applications that require real-time interactions, such as embodied AI and VR/AR. This paper explores a new strategy for camera-conditioned video generation of static scenes: using diffusion-based generative models to generate a sparse set of keyframes, and then synthesizing the full video through 3D reconstruction and rendering. By lifting keyframes into a 3D representation and rendering intermediate views, our approach amortizes the generation cost across hundreds of frames while enforcing geometric consistency. We further introduce a model that predicts the optimal number of keyframes for a given camera trajectory, allowing the system to adaptively allocate computation. Our final method, SRENDER, uses very sparse keyframes for simple trajectories and denser ones for complex camera motion. This results in video generation that is more than 40 times faster than the diffusion-based baseline in generating 20 seconds of video, while maintaining high visual fidelity and temporal stability, offering a practical path toward efficient and controllable video synthesis.

Efficient Camera-Controlled Video Generation of Static Scenes via Sparse Diffusion and 3D Rendering

TL;DR

SRENDER addresses the bottleneck of diffusion-based video generation by combining sparse, camera-conditioned keyframe diffusion with deterministic 3D Gaussian Splatting reconstruction to render dense video of static scenes. A transformer-based adaptive keyframe density predictor allocates computation based on trajectory complexity, and a two-stage diffusion process maintains coherence across sparse keyframes. Dense 3D reconstruction via AnySplat, followed by efficient 3DGS rendering and temporal chunking, yields substantial speedups (over 20–40×) while preserving or improving FID and FVD relative to strong diffusion baselines. The approach leverages explicit 3D reasoning to reduce computation dramatically, enabling real-time, controllable video synthesis for applications in embodied AI and VR/AR, with potential extension to dynamic scenes as 4D reconstruction advances mature.

Abstract

Modern video generative models based on diffusion models can produce very realistic clips, but they are computationally inefficient, often requiring minutes of GPU time for just a few seconds of video. This inefficiency poses a critical barrier to deploying generative video in applications that require real-time interactions, such as embodied AI and VR/AR. This paper explores a new strategy for camera-conditioned video generation of static scenes: using diffusion-based generative models to generate a sparse set of keyframes, and then synthesizing the full video through 3D reconstruction and rendering. By lifting keyframes into a 3D representation and rendering intermediate views, our approach amortizes the generation cost across hundreds of frames while enforcing geometric consistency. We further introduce a model that predicts the optimal number of keyframes for a given camera trajectory, allowing the system to adaptively allocate computation. Our final method, SRENDER, uses very sparse keyframes for simple trajectories and denser ones for complex camera motion. This results in video generation that is more than 40 times faster than the diffusion-based baseline in generating 20 seconds of video, while maintaining high visual fidelity and temporal stability, offering a practical path toward efficient and controllable video synthesis.
Paper Structure (56 sections, 5 equations, 5 figures, 4 tables, 1 algorithm)

This paper contains 56 sections, 5 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Camera-controlled Video Models
  4. 3D Reconstruction
  5. Keyframe Diffusion and 3D Rendering
  6. Adaptive Keyframe Selection
  7. Model.
  8. Supervision.
  9. Discussion.
  10. Keyframe Diffusion
  11. Preliminaries
  12. Diffusion Forcing and History Guidance.
  13. Our Model
  14. 3D Reconstruction and Rendering
  15. Preliminaries
  16. ...and 41 more sections

Figures (5)

  • Figure 1: Teaser. Left: Overview of our approach. Given an input image and a camera trajectory, SRENDER generates sparse keyframes, reconstructs the 3D scene, and renders the full video efficiently. Right: On average, our method is roughly 43 times faster than the history-guided video diffusion baseline (HG) song2025historyguidedvideodiffusion when generating 20-second 30-fps videos from the DL3DV dataset, achieving real-time performance while maintaining comparable or better video quality.
  • Figure 2: Overview of SRENDER. Given an image and a target camera trajectory, a keyframe density predictor predicts the optimal keyframe density for the depicted scene and camera trajectory. Keyframe poses are then uniformly sampled along the trajectory before being fed to the keyframe generation model together with the input image for generating keyframes. A 3D reconstruction model takes the keyframes and generates the 3D representation of the scene. Finally, the video is rendered from the 3D scene along the input camera trajectory.
  • Figure 3: Qualitative comparisons on DL3DV. All methods generate 20-second videos at 5 fps, conditioned on the input image and target trajectory. Output frames at 4s and 14s are visualized. Our method achieves both high video quality and camera control. HG song2025historyguidedvideodiffusion often has high-frequency artifacts. Voyager huang2025voyagerlongrangeworldconsistentvideo fails at generating a consistent long video sequence. We also show results with 2D interpolation methods huang2022rifereda2022film, which show strong morphing effects, and also cannot satisfy the intermediate camera control inputs.
  • Figure 4: Qualitative comparisons on RE10K. Compared with HG song2025historyguidedvideodiffusion, our method does not have high-frequency artifacts and is significantly faster.
  • Figure 5: Too few keyframes lead to visible holes in the generated video (red boxes), while generating too many keyframes is significantly more expensive, without significant quality gains. SRENDER selects the optimal number of keyframes (green) that strikes a good balance between completeness and efficiency.