4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion

TL;DR

4Real-Video tackles 4D video generation by recasting a frame-grid into two parallel token streams—temporal and view—that are updated independently via pre-trained transformer layers and synchronized through a dedicated layer. The method leverages hard or soft synchronization inspired by optimization theory, enabling efficient, high-quality 4D generation without explicit camera conditioning. Training uses velocity-matching on 2D-transformed data and a small Objaverse-based 4D set, achieving strong temporal and multi-view consistency while delivering substantial speed gains (approximate generation of an $8\times8$ grid at $288\times512$ in about 1 minute). It further demonstrates explicit 3D reconstructions via deformable Gaussian splats, validating coherent 3D structure alongside photorealistic 4D content. This approach advances data-efficient, controllable 4D content creation with practical implications for dynamic scenes and immersive applications.

Abstract

We propose 4Real-Video, a novel framework for generating 4D videos, organized as a grid of video frames with both time and viewpoint axes. In this grid, each row contains frames sharing the same timestep, while each column contains frames from the same viewpoint. We propose a novel two-stream architecture. One stream performs viewpoint updates on columns, and the other stream performs temporal updates on rows. After each diffusion transformer layer, a synchronization layer exchanges information between the two token streams. We propose two implementations of the synchronization layer, using either hard or soft synchronization. This feedforward architecture improves upon previous work in three ways: higher inference speed, enhanced visual quality (measured by FVD, CLIP, and VideoScore), and improved temporal and viewpoint consistency (measured by VideoScore and Dust3R-Confidence).

Figures (9)

• Figure 1: 4Real-Video is a 4D generation framework that (top-left) takes a fixed-view video and a freeze-time video as input and generates a grid of consistent video frames. One axis of the grid varies in time, and the other axis varies the viewpoint. The input videos can be real videos or videos generated by a video model. Note that our method can generate grids larger than $8 \times 8$ videos. Here, we present subsets of frames as an example. (top-right) 4D videos generated from generated videos. (bottom) We can also capture a real-world scene, and generate a 4D video given different prompts.
• Figure 2: Overview of 4Real-Video. Left: we initialize the grid of frames with a (generated or real) fixed-viewpoint video in the first row and a freeze-time video in the first column. Middle: our architecture consists of two parallel token streams. The top part processes $\mathbf{x}_l^\text{v}$ with viewpoint updates and the bottom part processes $\mathbf{x}_l^\text{t}$ with temporal updates. Subsequently, a synchronization layer computes the new tokens $\mathbf{x}_{l+1}^\text{v}$$\mathbf{x}_{l+1}^\text{t}$ for the next layer in the diffusion transformer architecture. Right: we propose two implementations of the synchronization layer: hard and soft synchronization.
• Figure 3: The dynamics of soft synchronization during inference.
• Figure 4: Visual Comparisons. We show two viewpoints for a fixed time for each method. Our method produces high-quality images, even under significant camera motion. In contrast, frames generated by 4Real and SV4D tend to appear more blurred, with objects notably distorted in SV4D. MotionCtrl struggles to generate frames under substantial camera motion. We use red bounding boxes to highlight regions with distortions and flickering, which become particularly noticeable when viewed as a video.
• Figure 5: Results from 4Real-Video. We can generate diverse and high-quality dynamic content.