AnyView: Synthesizing Any Novel View in Dynamic Scenes

TL;DR

The paper tackles the challenge of synthesizing consistent novel views from dynamic scenes using only monocular input. It introduces AnyView, a diffusion-based dynamic view synthesis framework that conditions on two camera viewpoints via Plücker ray conditioning to learn an implicit 4D representation from a large, diverse mix of 4D datasets. It also proposes AnyViewBench to rigorously evaluate extreme dynamic view synthesis across multiple domains, showing state-of-the-art zero-shot performance against strong baselines. The work enables realistic, temporally coherent novel-view video generation with broad potential applications in robotics, autonomous driving, and VR/AR, while reducing reliance on explicit 3D reconstruction or test-time optimization.

Abstract

Modern generative video models excel at producing convincing, high-quality outputs, but struggle to maintain multi-view and spatiotemporal consistency in highly dynamic real-world environments. In this work, we introduce \textbf{AnyView}, a diffusion-based video generation framework for \emph{dynamic view synthesis} with minimal inductive biases or geometric assumptions. We leverage multiple data sources with various levels of supervision, including monocular (2D), multi-view static (3D) and multi-view dynamic (4D) datasets, to train a generalist spatiotemporal implicit representation capable of producing zero-shot novel videos from arbitrary camera locations and trajectories. We evaluate AnyView on standard benchmarks, showing competitive results with the current state of the art, and propose \textbf{AnyViewBench}, a challenging new benchmark tailored towards \emph{extreme} dynamic view synthesis in diverse real-world scenarios. In this more dramatic setting, we find that most baselines drastically degrade in performance, as they require significant overlap between viewpoints, while AnyView maintains the ability to produce realistic, plausible, and spatiotemporally consistent videos when prompted from \emph{any} viewpoint. Results, data, code, and models can be viewed at: https://tri-ml.github.io/AnyView/

Figures (12)

• Figure 1: Enabling consistent extreme monocular dynamic view synthesis: We introduce AnyView, a diffusion framework that can generate videos of dynamic scenes from any chosen perspective, conditioned on a single input video. Our model operates end-to-end, without explicit scene reconstruction or expensive test-time optimization techniques. Existing methods tend to fail to extrapolate, largely copying the input view. More recent baselines can recover the overall structure in some cases (1st, 2nd rows), but fail when the camera trajectories become more complex (3rd row). Meanwhile, our method preserves scene geometry, appearance, and dynamics, despite working with drastically different target poses and highly "incomplete" visual observations. (D) indicates a baseline that relies on reprojected point clouds from estimated depth maps.
• Figure 2: The AnyView architecture. For both the clean input and noisy target videos, we concatenate pixels (RGB values) and camera information (Plücker vectors) belonging to the same viewpoint along the channel dimension, after independently encoding each modality into latent embeddings. We then stack these two multimodal videos along the sequence dimension, for a total of $2 \cdot t \cdot h \cdot w$ tokens, which are fed into the diffusion transformer to iteratively denoise the target video.
• Figure 3: Overview of our training data mixture. We train and evaluate AnyView on both single-view and multi-view videos from four domains: 3D, Driving, Robotics, and Other (see Section \ref{['sec:datasets']}). During training, we perform weighted sampling to ensure each domain is seen equally often, i.e. comprises 25% of the batch.
• Figure 4: AnyView in-domain DVS results on Kubric-4D (left) and Pardom-4D (right). We show the first and last frame of each video. The scene layout is generally preserved very well, despite drastic viewpoint changes and/or heavy occlusion from the input vantage point.
• Figure 5: Results on DyCheck iPhone (0-shot narrow DVS). While these scenes are not highly dynamic, they do contain subtle, intricate motions and hand-object interactions.