Generating Human Motion Videos using a Cascaded Text-to-Video Framework

TL;DR

CAMEO introduces a cascaded yet integrated text-to-video pipeline that couples a text-to-motion model with a conditioned video diffusion model to generate coherent human motion videos directly from text. The authors address training-time misalignment through a caption refinement strategy and SMPL-based motion cues, and they solve inference-time challenges with a camera-view selection module that aligns viewpoints to the input text. They demonstrate strong performance on MovieGen and a new HuMoBench benchmark, and show practical extensions such as motion and camera view editing. The work advances general human-centric video generation by enabling end-to-end T2V pipelines with improved motion fidelity, pose consistency, and viewpoint coherence. Limitations include reliance on the quality of the T2M backbone and single-person data, with potential for broader generalization as multi-person motion modeling improves.

Abstract

Human video generation is becoming an increasingly important task with broad applications in graphics, entertainment, and embodied AI. Despite the rapid progress of video diffusion models (VDMs), their use for general-purpose human video generation remains underexplored, with most works constrained to image-to-video setups or narrow domains like dance videos. In this work, we propose CAMEO, a cascaded framework for general human motion video generation. It seamlessly bridges Text-to-Motion (T2M) models and conditional VDMs, mitigating suboptimal factors that may arise in this process across both training and inference through carefully designed components. Specifically, we analyze and prepare both textual prompts and visual conditions to effectively train the VDM, ensuring robust alignment between motion descriptions, conditioning signals, and the generated videos. Furthermore, we introduce a camera-aware conditioning module that connects the two stages, automatically selecting viewpoints aligned with the input text to enhance coherence and reduce manual intervention. We demonstrate the effectiveness of our approach on both the MovieGen benchmark and a newly introduced benchmark tailored to the T2M-VDM combination, while highlighting its versatility across diverse use cases.

Figures (8)

• Figure 1: Qualitative advantage of CAMEO. Our approach, CAMEO produces more stable and consistent human body articulation in complex motions, whereas vanilla CogVideoX-5B yang2025cogvideox often shows pose distortion and inconsistent appearances. For instance, the vanilla model may generate implausible artifacts such as a character repeatedly picking up and putting down a phone. In contrast, our method maintains stable action continuity and prevents such inconsistencies.
• Figure 2: Overview of CAMEO. Given a text prompt, we first disentangle it to separate motion-related and semantic components. The motion prompt is converted into an initial motion sequence via a text-to-motion model. The sequence is rendered as SMPL-based guidance videos, where a camera-aware conditioning module determines the viewpoints for rendering. Finally, the video diffusion model synthesizes the human video, guided by the semantic prompt and motion condition, seamlessly bridging text-to-motion and text-to-video generation.
• Figure 2: Quantitative results for ablation studies.w/o Refine: ablated on text refinement; w/o View Module: ablated on view selection.
• Figure 3: Analysis of training choices. (a) The model trained with the original coarse captions often fails to learn fine-grained motion details, whereas the model trained with our refined captions converges faster and produces more accurate motion details. (b) Macro body movements emerge early in the denoising process, while finer details such as clearer body outlines appear later, around $t = 0.6$.
• Figure 4: Qualitative comparisons. Our pipeline captures complex human structures and motions more faithfully than baseline models, while also producing more natural and consistent camera views.