Beyond Global Alignment: Fine-Grained Motion-Language Retrieval via Pyramidal Shapley-Taylor Learning

TL;DR

This work tackles fine-grained motion-language retrieval by addressing the inadequacy of global alignment in capturing local joint- and segment-level interactions. It introduces Pyramidal Shapley-Taylor (PST), a three-level framework that progressively aligns motion and language through joint-wise, segment-wise, and holistic stages, guided by Shapley-Taylor Interaction (STI). An STI Estimation Head and a token compression pipeline (convolution, self-attention, and KNN-DPC) enable efficient, interpretable modeling of cross-modal interactions, with an InfoNCE-based objective augmented by STI distillation and self-distillation losses. Empirical results on HumanML3D and KIT-ML show state-of-the-art retrieval performance for both directions, with further gains when using LLM-enhanced part-level text descriptions (PST++), highlighting the method’s practical impact for fine-grained, controllable motion-language understanding.

Abstract

As a foundational task in human-centric cross-modal intelligence, motion-language retrieval aims to bridge the semantic gap between natural language and human motion, enabling intuitive motion analysis, yet existing approaches predominantly focus on aligning entire motion sequences with global textual representations. This global-centric paradigm overlooks fine-grained interactions between local motion segments and individual body joints and text tokens, inevitably leading to suboptimal retrieval performance. To address this limitation, we draw inspiration from the pyramidal process of human motion perception (from joint dynamics to segment coherence, and finally to holistic comprehension) and propose a novel Pyramidal Shapley-Taylor (PST) learning framework for fine-grained motion-language retrieval. Specifically, the framework decomposes human motion into temporal segments and spatial body joints, and learns cross-modal correspondences through progressive joint-wise and segment-wise alignment in a pyramidal fashion, effectively capturing both local semantic details and hierarchical structural relationships. Extensive experiments on multiple public benchmark datasets demonstrate that our approach significantly outperforms state-of-the-art methods, achieving precise alignment between motion segments and body joints and their corresponding text tokens. The code of this work will be released upon acceptance.

Figures (11)

• Figure 1: Overview of our Pyramidal Shapley-Taylor (PST) learning framework. Our PST learning framework consists of Shapley-Taylor Interaction (STI), described in Sec. \ref{['sec:sti']}, and pyramidal modeling scheme, described in Sec. \ref{['sec:plf']}. As illustrated in the middle cube, each cell represents the interaction strength between a motion token and a text token within a batch, where darker colors indicate stronger semantic correlations, and lighter colors represent weaker ones.
• Figure 2: An intuitive illustration of the STI.
• Figure 3: Visualization results of segment-wise alignment. We omit <EOS> for clarity and use commas to separate each individual word.
• Figure 4: Visualization results of joint-wise alignment. Darker colors indicate higher similarity scores.
• Figure 5: Qualitative results of text-to-motion retrieval.