PALUM: Part-based Attention Learning for Unified Motion Retargeting

TL;DR

PALUM tackles cross-skeleton motion retargeting by learning skeleton-agnostic representations via six semantic body parts and a spatio-temporal transformer. A cycle-consistent training regime preserves motion semantics while conditioning the retargeting on target topology, enabling robust generalization to unseen skeletons. The approach combines spatial attention within body parts, temporal dynamics modeling, and skeleton-specific decoding with T-pose and joint-name embeddings to achieve high-fidelity transfers. Experimental results on Mixamo demonstrate superior quantitative and qualitative performance, underscoring the method's practical impact for automated cross-character animation and asset reuse.

Abstract

Retargeting motion between characters with different skeleton structures is a fundamental challenge in computer animation. When source and target characters have vastly different bone arrangements, maintaining the original motion's semantics and quality becomes increasingly difficult. We present PALUM, a novel approach that learns common motion representations across diverse skeleton topologies by partitioning joints into semantic body parts and applying attention mechanisms to capture spatio-temporal relationships. Our method transfers motion to target skeletons by leveraging these skeleton-agnostic representations alongside target-specific structural information. To ensure robust learning and preserve motion fidelity, we introduce a cycle consistency mechanism that maintains semantic coherence throughout the retargeting process. Extensive experiments demonstrate superior performance in handling diverse skeletal structures while maintaining motion realism and semantic fidelity, even when generalizing to previously unseen skeleton-motion combinations. We will make our implementation publicly available to support future research.

Figures (5)

• Figure 1: Overview of our motion retargeting framework. (a) Encoder-decoder architecture: The transformer encoder processes source motion sequences through multiple attention pooling layers to extract skeleton-agnostic motion representations. These representations are fed to an MLP that outputs key-value pairs for cross-attention in the transformer decoder. The decoder takes uniformly sampled noise and skeleton-specific embeddings to generate target motion sequences. Note that the target skeleton shown in this pipeline includes pauldron joints, demonstrating our method's capability to handle diverse skeletal topologies. (b) Training pipeline: The model is trained using reconstruction and cycle consistency objectives (additional loss terms are detailed in Section \ref{['subsec::train_objective']}), where Motion A from Skeleton A is retargeted to Skeleton B, then back to Skeleton A, ensuring motion preservation across different skeleton structures. (c) Testing pipeline: During inference, only the forward retargeting path is used, with the cycle consistency components (marked with red crosses) disabled, allowing direct motion transfer from the source to the target skeleton.
• Figure 2: T-pose examples after our joint elimination strategy. (1) Warrok: the pauldron joints are named as "RightArmourx" (x=1,2,3,4,5) in the BVH file, so they match our "RightArm" joint name selection and they are preserved. (2) BigVegas: the hair joints are named as "HeadTop_Endx" (x=1,2), which match our "Head" joint name selection, so they are preserved.
• Figure 3: Qualitative results of our method and the baselines. Our method preserves the natural motion dynamics and joint relationships.
• Figure 4: Qualitative results of ablation studies demonstrating the impact of key design components. We show overlaid predictions with the GT target. Without shared joints, body parts move independently, causing unnatural artifacts such as extreme head tilting. Excluding positional information reduces end-effector accuracy, while joint masking disrupts hierarchical skeletal relationships, leading to degraded motion quality.
• Figure 5: Mixamo motion retargeted to other skeletons. Source motion from the AJ skeleton is retargeted to SMPL, MetaHuman, and MetaHuman Merged.