3D-Aware Implicit Motion Control for View-Adaptive Human Video Generation

TL;DR

This work tackles view-adaptive human video generation by learning a 3D-aware implicit motion representation that remains decoupled from explicit camera control. It introduces 3DiMo, which jointly trains a motion encoder with a pretrained DiT-based video generator, injecting compact motion tokens through cross-attention to realize 3D-aware, text-guided synthesis. Training leverages view-rich data (single-view, multi-view, moving-camera) and uses lightweight geometric supervision that is gradually annealed, allowing the model to acquire genuine 3D spatial understanding from data and generator priors. Experiments show that 3DiMo achieves superior motion fidelity and visual quality compared to 2D- and SMPL-based baselines, while maintaining flexible camera control and robust 3D consistency across viewpoints.

Abstract

Existing methods for human motion control in video generation typically rely on either 2D poses or explicit 3D parametric models (e.g., SMPL) as control signals. However, 2D poses rigidly bind motion to the driving viewpoint, precluding novel-view synthesis. Explicit 3D models, though structurally informative, suffer from inherent inaccuracies (e.g., depth ambiguity and inaccurate dynamics) which, when used as a strong constraint, override the powerful intrinsic 3D awareness of large-scale video generators. In this work, we revisit motion control from a 3D-aware perspective, advocating for an implicit, view-agnostic motion representation that naturally aligns with the generator's spatial priors rather than depending on externally reconstructed constraints. We introduce 3DiMo, which jointly trains a motion encoder with a pretrained video generator to distill driving frames into compact, view-agnostic motion tokens, injected semantically via cross-attention. To foster 3D awareness, we train with view-rich supervision (i.e., single-view, multi-view, and moving-camera videos), forcing motion consistency across diverse viewpoints. Additionally, we use auxiliary geometric supervision that leverages SMPL only for early initialization and is annealed to zero, enabling the model to transition from external 3D guidance to learning genuine 3D spatial motion understanding from the data and the generator's priors. Experiments confirm that 3DiMo faithfully reproduces driving motions with flexible, text-driven camera control, significantly surpassing existing methods in both motion fidelity and visual quality.

Figures (6)

• Figure 1: 3DiMo can faithfully reproduce the 3D spatial motion from a 2D driving video, supporting flexible text-guided camera control.
• Figure 2: Overview of 3DiMo. Our framework consists of end-to-end trained motion encoders---$\mathcal{E}_b$ for the body and $\mathcal{E}_h$ for hands---and an DiT-based video generator. Given a reference frame $\bm{I}_R$ and a driving video $\bm{V}_D$, driving frames are first augmented with random perspective transformations before being encoded by the motion encoder to extract view-agnostic motion representations. These resulting features are then injected into the generator through cross-attention, enabling the model to synthesize a target sequence $\bm{V}_{\mathrm{tgt}}$ that reenacts the same underlying 3D motion while preserving flexible text-driven camera control. To facilitate 3D-aware learning, we introduce early-stage auxiliary geometric supervision by regressing the encoded motion to external parametric reconstruction results $\theta_b$ and $\theta_h$. During training, view-rich data is used to jointly supervise same-view reconstruction and cross-view motion reproduction, driving the emergence of expressive and 3D-aware motion representations. At inference, motion tokens extracted directly from 2D driving frames provide rich 3D spatial cues that can animate any reference character, supporting high-fidelity and view-adaptive motion-controlled video generation.
• Figure 3: Our collected view-rich dataset combines internet videos, UE renderings, and in-house captures, covering camera categories including single-view, multi-view, and camera-motion sequences. High-quality large-scale single-view footage exposes the model to diverse human motions, while complementary multi-view data provides consistent cross-view observations that are crucial for learning genuine 3D-aware motion representations.
• Figure 4: Visualization comparisons with baselines. Red and yellow bounding boxes highlight depth ambiguities and inaccurate poses, respectively. "A.A." denotes AnimateAnyone. Our method produces accurate and 3D-plausible motion reenactment videos.
• Figure 5: Visualizations of ablation results. Using SMPL poses as motion representation introduces typical depth ambiguity errors. Removing any view-rich data supervision impairs camera control. Removing auxiliary geometric supervision or using channel concatenation causes training instability and quality degradation. Without the hand encoder, fine-grained hand motions are lost.