SUGAR: Learning Skeleton Representation with Visual-Motion Knowledge for Action Recognition

TL;DR

SUGAR tackles skeleton-based action recognition by grounding skeletal signals in rich visual and motion priors and utilizing large language models as recognizers. A three-stage pipeline first constructs transferable text knowledge from motion and visuals, then learns discrete skeleton representations via a vision-language framework with MIL-based contrastive learning, and finally maps these representations into an LLM's token space using Temporal Query Projection, with LoRA fine-tuning. The approach yields state-of-the-art or competitive results on Toyota Smarthome, NTU 60/120, and PKU-MMD, and demonstrates strong zero-shot generalization across datasets and unseen actions. This work highlights the practicality of combining visual-language priors with LLMs for more discriminative, language-grounded action understanding and description in real-world scenarios.

Abstract

Large Language Models (LLMs) hold rich implicit knowledge and powerful transferability. In this paper, we explore the combination of LLMs with the human skeleton to perform action classification and description. However, when treating LLM as a recognizer, two questions arise: 1) How can LLMs understand skeleton? 2) How can LLMs distinguish among actions? To address these problems, we introduce a novel paradigm named learning Skeleton representation with visUal-motion knowledGe for Action Recognition (SUGAR). In our pipeline, we first utilize off-the-shelf large-scale video models as a knowledge base to generate visual, motion information related to actions. Then, we propose to supervise skeleton learning through this prior knowledge to yield discrete representations. Finally, we use the LLM with untouched pre-training weights to understand these representations and generate the desired action targets and descriptions. Notably, we present a Temporal Query Projection (TQP) module to continuously model the skeleton signals with long sequences. Experiments on several skeleton-based action classification benchmarks demonstrate the efficacy of our SUGAR. Moreover, experiments on zero-shot scenarios show that SUGAR is more versatile than linear-based methods.

Figures (7)

• Figure 1: Comparison between existing action recognition paradigms and our SUGAR.
• Figure 2: Overall framework of SUGAR. The complete training procedure is divided into three parts. We use the GPT-generated fine-grained action description and VLM-generated visual description as input for the text encoder to supervise the skeleton representation learning, where the linear layer maps the skeleton to the same feature space as the text. During inference, only the skeleton data needs to be input for action recognition.
• Figure 3: Temporal Query Projection consists of a number of Q-Formers blip2 and a linear layer.
• Figure 4: The effect of the length of action tokens on NTU RGB+D, where 1000 denotes the length of the complete time series output by the skeleton encoder and 1 represents action tokens after temporal compression.
• Figure 5: Before training