MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Second

TL;DR

<3-5 sentence high-level summary>MoVieS tackles the challenge of reconstructing dynamic 3D scenes and rendering novel views from monocular video in real time. It introduces dynamic splatter pixels—time-varying Gaussian primitives—and a motion head to jointly model appearance, geometry, and motion within a single feed-forward framework. Trained on diverse datasets with curriculum learning, it achieves competitive 4D perception performance and orders-of-magnitude speedups over optimization-based methods, while enabling zero-shot scene flow and moving object segmentation. This approach broadens the practicality of dynamic scene understanding for robotics, AR/VR, and digital twins.

Abstract

We present MoVieS, a novel feed-forward model that synthesizes 4D dynamic novel views from monocular videos in one second. MoVieS represents dynamic 3D scenes using pixel-aligned grids of Gaussian primitives, explicitly supervising their time-varying motion. This allows, for the first time, the unified modeling of appearance, geometry and motion, and enables view synthesis, reconstruction and 3D point tracking within a single learning-based framework. By bridging novel view synthesis with dynamic geometry reconstruction, MoVieS enables large-scale training on diverse datasets with minimal dependence on task-specific supervision. As a result, it also naturally supports a wide range of zero-shot applications, such as scene flow estimation and moving object segmentation. Extensive experiments validate the effectiveness and efficiency of MoVieS across multiple tasks, achieving competitive performance while offering several orders of magnitude speedups.

Figures (9)

• Figure 1: Overview. MoVieS consists of a shared image encoder, an attention-based feature backbone (Section \ref{['subsubsec:backbone']}), and three heads (Section \ref{['subsubsec:heads']}) that simultaneously predict appearance, geometry and motion. Image shortcut for splatter head and time-varying Gaussian attributes are omitted for brevity.
• Figure 2: Motion Head. Given $t_q$ target timesteps, the proposed motion head is conditioned via adaptive layer normalization (AdaLN) and predicts 3D displacements for each input pixel. After rasterization using the $M$ corresponding query-time cameras, output images in shape $M\times 3\times H\times W$ are rendered for supervision. Gaussian attribute deformation $\Delta\mathbf{a}$ is omitted for brevity.
• Figure 3: Novel View Synthesis for Dynamic Scenes. Given a monocular video, we compare synthesized views from a novel view across different methods. Regions invisible in the input are rendered as black or white, depending on the rendering implementation. More results in Figure \ref{['fig:nvs']}.
• Figure 4: Motion Visualization for Ablation Studies. We investigate key factors affecting motion learning in MoVieS, such as loss design and synergy with view synthesis. XYZ values in motion maps are normalized as RGB for visualization. Red arrows on video frames indicate motion directions.
• Figure 5: Zero-shot Applications. The predicted pixel-aligned motion maps from our model can be directly applied to downstream tasks, such as (a) scene flow estimation and (b) moving object segmentation, in a zero-shot manner, without any task-specific fine-tuning or supervision.