Human4DiT: 360-degree Human Video Generation with 4D Diffusion Transformer

TL;DR

Human4DiT introduces a 4D diffusion transformer to generate 360-degree human videos from a single image by modeling view, time, and spatial correlations through cascaded 2D image, view, and temporal transformers. The model integrates CNN-based conditioning for identity, SMPL-based motion, temporal cues, and camera parameters to enable controllable, high-fidelity generation across monocular, multi-view, 3D static, and 360-degree scenarios. A multi-dimensional dataset and two-stage spatio-temporal sampling strategy enable training and inference of long, coherent videos under window constraints. Extensive experiments demonstrate superior versatility and coherence over state-of-the-art methods, with ablations confirming the critical roles of temporal and view transformers and efficient sampling. The work advances 4D content generation with practical implications for VR, animation, and interactive media, while also addressing ethical considerations around misuse.

Abstract

We present a novel approach for generating 360-degree high-quality, spatio-temporally coherent human videos from a single image. Our framework combines the strengths of diffusion transformers for capturing global correlations across viewpoints and time, and CNNs for accurate condition injection. The core is a hierarchical 4D transformer architecture that factorizes self-attention across views, time steps, and spatial dimensions, enabling efficient modeling of the 4D space. Precise conditioning is achieved by injecting human identity, camera parameters, and temporal signals into the respective transformers. To train this model, we collect a multi-dimensional dataset spanning images, videos, multi-view data, and limited 4D footage, along with a tailored multi-dimensional training strategy. Our approach overcomes the limitations of previous methods based on generative adversarial networks or vanilla diffusion models, which struggle with complex motions, viewpoint changes, and generalization. Through extensive experiments, we demonstrate our method's ability to synthesize 360-degree realistic, coherent human motion videos, paving the way for advanced multimedia applications in areas such as virtual reality and animation.

Figures (10)

• Figure 1: We propose Human4DiT, a novel approach to generate 360-degree high-quality, spatio-temporally coherent human videos given a reference image. With the proposed 4D diffusion transformer, our method is capable of generating monocular video, multi-view video, 3D static video, and 360-degree rotating video.
• Figure 2: Pipeline of Human4DiT: our framework is based on 4D diffusion transformer, which adopts a cascaded structure consisting of the 2D image transformer, the view transformer, and the temporal transformer. The input contains a reference image $\mathbf{y}_{r}$, dynamic SMPL sequences $\mathbf{P}$, and camera parameters $\mathbf{c}$. Starting from a generated noisy latent representation $\mathbf{z}_{t}$, we denoise them conditioned on $\mathbf{y}_{r}$, $\mathbf{P}$, and $\mathbf{c}$ to recover the original latent frames. First, the 2D image transformer block is designed to capture spatial self-attention within latent frame tokens and pose frame tokens extracted by latent encoder $\mathbf{E}_\mathbf{z}$ and pose encoder $\mathbf{E}_{p}$, respectively. In addition, ID tokens $\mathbf{y}_{id}$ and image embedding $\mathbf{y}_e$ extracted from $\mathbf{y}_{r}$ by ID encoder $\mathbf{E}_{id}$ and CLIP are also injected to ensure identity consistency. Secondly, we use the view transformer block to learn correspondences across different viewpoints conditioned on camera embedding. Finally, we adopt a temporal transformer to capture temporal correlations with time embedding. The time embedding and camera embedding are obtained by positional encoding time $\mathbf{T}$ and camera $\mathbf{c}$, respectively.$^5$
• Figure 3: Human4DiT dataset: In addition to the open-source dataset, we collect a multi-dimension dataset including 10k monocular videos from Internet, 5k high-quality 3D human scans and 100 animatable human models for dynamic free-view rendering.
• Figure 4: Qualitative comparison on monocular video.
• Figure 5: Ablation study of spatial-temporal sampling.