Pixel-to-4D: Camera-Controlled Image-to-Video Generation with Dynamic 3D Gaussians

TL;DR

Pixel-to-4D tackles camera-controlled image-to-video generation by introducing a 4D scene representation based on per-pixel Gaussian splats with linear and angular velocities and accelerations. The method predicts static Gaussian parameters from a single image and augment them with generative motion conditioned on image features and a variational latent, enabling coherent future-frame rendering along user-defined camera paths in a single forward pass. A differentiable Gaussian rasterizer renders future frames with depth refinement guided by Depth-Pro, while per-object motion sharing enforces coherent object dynamics. Evaluations on KITTI, Waymo, RealEstate10K, and DL3DV-10K show state-of-the-art video quality (PSNR, SSIM, LPIPS, FVD) and faster inference compared to baselines, demonstrating robust camera controllability and temporal/geometric consistency for real-world urban scenes.

Abstract

Humans excel at forecasting the future dynamics of a scene given just a single image. Video generation models that can mimic this ability are an essential component for intelligent systems. Recent approaches have improved temporal coherence and 3D consistency in single-image-conditioned video generation. However, these methods often lack robust user controllability, such as modifying the camera path, limiting their applicability in real-world applications. Most existing camera-controlled image-to-video models struggle with accurately modeling camera motion, maintaining temporal consistency, and preserving geometric integrity. Leveraging explicit intermediate 3D representations offers a promising solution by enabling coherent video generation aligned with a given camera trajectory. Although these methods often use 3D point clouds to render scenes and introduce object motion in a later stage, this two-step process still falls short in achieving full temporal consistency, despite allowing precise control over camera movement. We propose a novel framework that constructs a 3D Gaussian scene representation and samples plausible object motion, given a single image in a single forward pass. This enables fast, camera-guided video generation without the need for iterative denoising to inject object motion into render frames. Extensive experiments on the KITTI, Waymo, RealEstate10K and DL3DV-10K datasets demonstrate that our method achieves state-of-the-art video quality and inference efficiency. The project page is available at https://melonienimasha.github.io/Pixel-to-4D-Website.

Figures (4)

• Figure 1: Pixel-to-4D: Given an input image $I_t$, $\mathrm{enc}_s$ encodes $I_t$ and its estimated depths $D_t$ and fuses features from DINOv2. The combined features are decoded by $\mathrm{dec_s}$ to predict static Gaussian parameters $d,\Delta,r,s,\sigma,c$. Conditioned on the combined features, splat velocities $v$ and accelerations $a$ are generated using $\mathrm{dec_{vae}}$ and $\mathrm{dec_d}$ from latent Gaussian noise. These are aggregrated over object segmentations to give final linear and angular velocities and accelerations. Then, a set of Gaussians $G_{t+ \delta t}$ are derived from the static Gaussian parameters and velocities and accelerations, for a future time $t+ \delta t$, from which the future frame is rendered with relative camera pose $\delta\pi$. The model is supervised by ground-truth future frames and their estimated depth-maps. During training, the inputs and model components within the blue box, $I_{t+1}$, $D_{t+1}$, $L_{t+1}$, $\mathrm{enc_d}$,$\mathrm{conv_d}$ and $\mathrm{enc_{vae}}$, are used to reconstruct $z$, and the model is optimized to align $z \sim \mathcal{N}(0, I)$. The green arrows represent skip connections from $\mathrm{enc}_s$ to $\mathrm{dec_s}$ and $\mathrm{enc_d}$.
• Figure 2: Qualitative comparisons on four datasets. Each block shows the input frame at $t=0$ and ground truth and generated results at $t+0.5s$, $t+1s$, and $t+1.5s$ for ours and four baseline methods. Our method has shown the best camera-controllability and visual-quality.
• Figure 3: Qualitative ablation results on Waymo: Input and predicted frames and depths at $t+0.9s$. The model with velocities has succesfully captured the motion of the black car.
• Figure 4: Qualitative ablation results on KITTI, showing input and predicted frames and depths at $t+0.9s$. Top: with deterministic velocities and generative velocities. Motion of the black car is more realistic in gen. velocities version. Middle: without DINOv2 and with DINOv2. Appearance of the black car is more realistic in with DINOv2. Bottom: with 1 Gaussian per pixel and 5 Gaussians per pixel. Around nearby trees and the road are blurry in the 1-Gaussian version.