Plenoptic Video Generation

TL;DR

PlenopticDreamer tackles multi-view plenoptic video re-rendering by enforcing long-term spatio-temporal memory within an autoregressive, multi-in-single-out diffusion model. It combines a Flow-based Video Diffusion Transformer with 3D FOV–guided video retrieval and Plücker raymap–driven camera conditioning, augmented by progressive context-scaling and self-conditioning to stabilize long-horizon generation. The approach delivers superior view synchronization, accurate camera control, and high-fidelity visuals across diverse camera transformations and long sequences, validated on Basic and Agibot benchmarks. This memory-driven framework advances practical plenoptic video generation for immersive and robotic manipulation tasks, while outlining limitations and future directions in handling complex motions and extreme long-shot regimes.

Abstract

Camera-controlled generative video re-rendering methods, such as ReCamMaster, have achieved remarkable progress. However, despite their success in single-view setting, these works often struggle to maintain consistency across multi-view scenarios. Ensuring spatio-temporal coherence in hallucinated regions remains challenging due to the inherent stochasticity of generative models. To address it, we introduce PlenopticDreamer, a framework that synchronizes generative hallucinations to maintain spatio-temporal memory. The core idea is to train a multi-in-single-out video-conditioned model in an autoregressive manner, aided by a camera-guided video retrieval strategy that adaptively selects salient videos from previous generations as conditional inputs. In addition, Our training incorporates progressive context-scaling to improve convergence, self-conditioning to enhance robustness against long-range visual degradation caused by error accumulation, and a long-video conditioning mechanism to support extended video generation. Extensive experiments on the Basic and Agibot benchmarks demonstrate that PlenopticDreamer achieves state-of-the-art video re-rendering, delivering superior view synchronization, high-fidelity visuals, accurate camera control, and diverse view transformations (e.g., third-person to third-person, and head-view to gripper-view in robotic manipulation). Project page: https://research.nvidia.com/labs/dir/plenopticdreamer/

Figures (16)

• Figure 1: We present PlenopticDreamer, a generative framework that re-renders input video under novel camera trajectories while preserving long-term spatio-temporal memory in hallucinated regions across overlapping views, thereby producing coherent plenoptic functions (see robot’s right side, highlighted in red dashed boxes across three trajectories). Please refer to our https://research.nvidia.com/labs/dir/plenopticdreamer/ for more results.
• Figure 2: PlenopticDreamer Framework. Its core is an autoregressive multi-camera video generator that retrieves $k$ video–camera pairs $\{(\mathbf{P}^n, \mathbf{V}^n)\}_{n=1}^k$ from the memory bank using a 3D FOV–based retrieval strategy. Conditioned on these retrieved pairs and the target camera $\mathbf{P}^{k+1}$, the model performs noisy scheduling and learnable reconstruction to generate the target video $\mathbf{V}^{k+1}$. To enable long video generation, a portion of the preceding frames in $\mathbf{V}^{k+1}$ is preserved as clean inputs at a certain ratio during training. Within each DiT block, temporal concatenation is applied to form video tokens $\mathbf{x}$ as in-context condition.
• Figure 3: FOV-based Retrieval Comparison. Unlike prior frame-level retrieval methods yu2025contextxiao2025worldmem, ours computes robust video-level similarity by averaging frame-wise similarities.
• Figure 4: Qualitative Comparison on the Basic Benchmark. PlenopticDreamer generates high-fidelity visuals with consistent hallucinations from different camera trajectories. In contrast, ReCamMaster and TrajectoryCrafter fail to preserve spatio-temporal consistency while maintaining visual quality, especially under large-angle viewpoint changes, such as leftward azimuth shifts.
• Figure 5: Qualitative Results on the Agibot Benchmark. Given a head-view manipulation video in Agibot, PlenopticDreamer-agibot (Ours) can generate temporally consistent videos from the left and right gripper viewpoints.