FUSION: Full-Body Unified Motion Prior for Body and Hands via Diffusion

TL;DR

FUSION presents the first unconditional full-body motion prior that jointly models body and detailed hand dynamics by unifying heterogeneous hand- and body-motion datasets into a single diffusion framework. It introduces a canonicalized trajectory representation for SMPL-X, a diffusion formulation with reconstruction, geometry, and foot-skating losses, and a diffusion-noise optimization pipeline to enforce task-specific constraints from diverse sources. Empirical results show competitive Keypoint Tracking performance and superior motion realism relative to skeletal baselines, while enabling applications in Human–Object Interaction and Self-Interaction guided by LLMs. The approach demonstrates that data fusion across modalities, combined with flexible optimization, yields plausible, fine-grained full-body motions and controllable hand articulation suitable for robotics, animation, and interactive systems.

Abstract

Hands are central to interacting with our surroundings and conveying gestures, making their inclusion essential for full-body motion synthesis. Despite this, existing human motion synthesis methods fall short: some ignore hand motions entirely, while others generate full-body motions only for narrowly scoped tasks under highly constrained settings. A key obstacle is the lack of large-scale datasets that jointly capture diverse full-body motion with detailed hand articulation. While some datasets capture both, they are limited in scale and diversity. Conversely, large-scale datasets typically focus either on body motion without hands or on hand motions without the body. To overcome this, we curate and unify existing hand motion datasets with large-scale body motion data to generate full-body sequences that capture both hand and body. We then propose the first diffusion-based unconditional full-body motion prior, FUSION, which jointly models body and hand motion. Despite using a pose-based motion representation, FUSION surpasses state-of-the-art skeletal control models on the Keypoint Tracking task in the HumanML3D dataset and achieves superior motion naturalness. Beyond standard benchmarks, we demonstrate that FUSION can go beyond typical uses of motion priors through two applications: (1) generating detailed full-body motion including fingers during interaction given the motion of an object, and (2) generating Self-Interaction motions using an LLM to transform natural language cues into actionable motion constraints. For these applications, we develop an optimization pipeline that refines the latent space of our diffusion model to generate task-specific motions. Experiments on these tasks highlight precise control over hand motion while maintaining plausible full-body coordination. The code will be public.

Figures (14)

• Figure 1: Our method, FUSION, enables full-body motion synthesis, including detailed hand motion. Using our optimization framework, we achieve realistic motions for Keypoint Tracking (left), Human--Object Interaction (middle), and Self-Interaction (right) tasks. The initial frames have lighter colors and they get darker over time. To see the motions please watch the supplementary video.
• Figure 2: Method Overview: Our framework (top) refines diffusion noise through an optimization process that incorporates multiple control signals (middle, bottom). These include example grasps (middle-right), where the difference between human hand joints (pink dots) and grasping hand joints (green dots) guides the optimization. Additionally, joint-based control signals (middle-left picture, green points) serve as loss functions to enforce precise motion constraints. Finally, high-level plans from an LLM (bottom) enable motion synthesis that adheres to specified contact vertices and timesteps, ensuring task-driven and contextually appropriate interactions. For each application our framework leverages backpropagation through the motion denoiser $M$ to iteratively update the noise. After optimization, the refined noise $X_{T}$ is decoded to produce the final motion sequence.
• Figure 3: FUSION qualitative results: FUSION can track any given joint location (all $10$ fingertips) over time (a). Conform a grasp provided by an off-the-shelf method(b) or Self-Interaction (c) In this case, it tracks the fingertips. Objects are provided only for visualization purposes, they do not play any role in the input. Mesh colors darken over time.
• Figure 4: FUSION failure cases. When the trajectory is highly dynamic, FUSION may fail to comply with task-specific requirements, which can lead to self-penetration and unrealistic motion. The example above illustrates this for the Keypoint Tracking task.
• Figure 5: Subject preference ratios on test split. Subjects were tasked with selecting one of three options: preferring FUSION, reporting no visible difference, and preferring the other method.