CoMoVi: Co-Generation of 3D Human Motions and Realistic Videos

TL;DR

CoMoVi tackles the intertwined problem of generating 3D human motions and realistic 2D videos by coupling two diffusion models in a single loop. It introduces a novel 2D motion representation that encodes 3D surface normals and body-part semantics to leverage pre-trained video diffusion priors, and a dual-branch diffusion with mutual feature interactions and 3D-2D cross-attention to jointly synthesize motion and video. A curated CoMoVi Dataset provides real-world video data with text and motion annotations to support training and evaluation. Experiments show strong motion fidelity and video realism, outperforming state-of-the-art text-to-motion and image-to-video baselines and demonstrating generalization to unseen data. Overall, CoMoVi offers a principled framework for synchronized co-generation of 3D human motion and photorealistic video with practical implications for animation, AR/VR, and gaming.

Abstract

In this paper, we find that the generation of 3D human motions and 2D human videos is intrinsically coupled. 3D motions provide the structural prior for plausibility and consistency in videos, while pre-trained video models offer strong generalization capabilities for motions, which necessitate coupling their generation processes. Based on this, we present CoMoVi, a co-generative framework that couples two video diffusion models (VDMs) to generate 3D human motions and videos synchronously within a single diffusion denoising loop. To achieve this, we first propose an effective 2D human motion representation that can inherit the powerful prior of pre-trained VDMs. Then, we design a dual-branch diffusion model to couple human motion and video generation process with mutual feature interaction and 3D-2D cross attentions. Moreover, we curate CoMoVi Dataset, a large-scale real-world human video dataset with text and motion annotations, covering diverse and challenging human motions. Extensive experiments demonstrate the effectiveness of our method in both 3D human motion and video generation tasks.

Figures (11)

• Figure 1: Different paradigms of motion video co-generation.
• Figure 1: Curation pipeline of our CoMoVi Dataset.
• Figure 2: We compress normals and body part semantics of 3D SMPL meshes into RGB images.
• Figure 2: Prompt instruction for Qwen3 qwen3 to analyze dense video captions.
• Figure 3: Pipeline overview of CoMoVi. Our method consists of an effective 2D human motion representation (Sec. \ref{['sec:morep']}) to encode 3D motion information in pixel space, and a dual-branch diffusion model extended from Wan2.2-I2V-5B to coordinate 2D motion and RGB video sequence denoising process with 3D-2D cross-attention modules to concurrently generate 3D human motion (Sec. \ref{['sec:comovi']}).