MotionDuet: Dual-Conditioned 3D Human Motion Generation with Video-Regularized Text Learning

TL;DR

MotionDuet tackles the challenge of producing realistic and controllable 3D human motion from text and video inputs by introducing a dual-conditioned diffusion framework. It fuses video-grounded priors and semantic text cues through the DUET module, enforces distributional and structural alignment with a DASH loss, and balances modalities with an auto-guidance mechanism that avoids manual tuning. Experiments on HumanML3D show MotionDuet achieves strong realism and controllability, with robust performance even in text-only inference due to video regularization. By transferring real-world spatio-temporal priors into the motion latent space, this approach enhances temporal coherence and semantic fidelity for multimodal motion generation with practical implications for animation, robotics, and embodied AI.

Abstract

3D Human motion generation is pivotal across film, animation, gaming, and embodied intelligence. Traditional 3D motion synthesis relies on costly motion capture, while recent work shows that 2D videos provide rich, temporally coherent observations of human behavior. Existing approaches, however, either map high-level text descriptions to motion or rely solely on video conditioning, leaving a gap between generated dynamics and real-world motion statistics. We introduce MotionDuet, a multimodal framework that aligns motion generation with the distribution of video-derived representations. In this dual-conditioning paradigm, video cues extracted from a pretrained model (e.g., VideoMAE) ground low-level motion dynamics, while textual prompts provide semantic intent. To bridge the distribution gap across modalities, we propose Dual-stream Unified Encoding and Transformation (DUET) and a Distribution-Aware Structural Harmonization (DASH) loss. DUET fuses video-informed cues into the motion latent space via unified encoding and dynamic attention, while DASH aligns motion trajectories with both distributional and structural statistics of video features. An auto-guidance mechanism further balances textual and visual signals by leveraging a weakened copy of the model, enhancing controllability without sacrificing diversity. Extensive experiments demonstrate that MotionDuet generates realistic and controllable human motions, surpassing strong state-of-the-art baselines.

Figures (11)

• Figure 1: MotionDuet is a multimodal framework for generating high-quality, controllable human motion under diverse conditions, including text prompts, video references, or their combination. Video results are provided in the supplementary material.
• Figure 2: MotionDuet framework overview. It primarily consists of three key steps: 1) fine-tuning video motion dataset based on a pre-trained model and freezing the weights to focus on inference (orange background); 2) proposing a dual-stream control mechanism combined with auto-guidance mechanism to integrate video and text inputs, effectively guiding motion generation (blue background); and 3) utilizing the DUET module (purple dashed box) combined with DASH Loss to align and fuse multimodal information, enhancing overall information processing capabilities.
• Figure 3: Qualitative results. MotionDuet captures motion direction and temporal coherence more accurately than prior methods, more results can be seen in Appendix \ref{['More_com_result']}. MoMask uses parallel masked modeling, while MLD adopts progressive diffusion denoising. In both rows, MotionDuet achieves smoother coordination and more precise dynamics. ${\dag}$ denotes text-only inference without video guidance.
• Figure 4: Qualitative results of model-generated motions for real-world videos involving complex actions. Examples include ballet spins and baseball pitching. In the golf swing sequence, the generated motion accurately captures the smooth and continuous rotation of the torso. In the baseball throwing example, the model vividly depicts the dynamic coordination between body rotation and arm extension, effectively conveying the power and fluidity of the motion. Additional qualitative results are provided in the Appendix \ref{['Qualitative Evaluation of Generalization on Unseen Real-World Videos']} .
• Figure 5: Qualitative experimental results. These examples cover a variety of challenging textual descriptions, involving complex action compositions and directional changes. MotionDuet is capable of generating motion sequences at a rate of approximately 199.61 poses per second during inference.