SkeletonAgent: An Agentic Interaction Framework for Skeleton-based Action Recognition

TL;DR

SkeletonAgent introduces an interactive, agent-based framework that links a skeleton-based action recognizer with an LLM via two agents, Questioner and Selector, to deliver context-aware discriminative guidance and fine-grained joint-level constraints. The system uses online performance feedback to craft targeted prompts and extract salient joints, achieving precise cross-modal alignment through both local constraints and bi-directional CLIP-based semantics. Extensive experiments across five benchmarks demonstrate consistent state-of-the-art results, including strong gains on challenging and fine-grained actions. The work highlights the value of dynamic, multi-turn LLM guidance for improving fine-grained skeleton-based action recognition.

Abstract

Recent advances in skeleton-based action recognition increasingly leverage semantic priors from Large Language Models (LLMs) to enrich skeletal representations. However, the LLM is typically queried in isolation from the recognition model and receives no performance feedback. As a result, it often fails to deliver the targeted discriminative cues critical to distinguish similar actions. To overcome these limitations, we propose SkeletonAgent, a novel framework that bridges the recognition model and the LLM through two cooperative agents, i.e., Questioner and Selector. Specifically, the Questioner identifies the most frequently confused classes and supplies them to the LLM as context for more targeted guidance. Conversely, the Selector parses the LLM's response to extract precise joint-level constraints and feeds them back to the recognizer, enabling finer-grained cross-modal alignment. Comprehensive evaluations on five benchmarks, including NTU RGB+D, NTU RGB+D 120, Kinetics-Skeleton, FineGYM, and UAV-Human, demonstrate that SkeletonAgent consistently outperforms state-of-the-art benchmark methods. The code is available at https://github.com/firework8/SkeletonAgent.

Figures (6)

• Figure 1: Comparison of (a) conventional multi-modal skeleton-based action recognition pipeline and (b) our SkeletonAgent. SkeletonAgent incorporates two cooperative agents, i.e., Questioner and Selector, that create an interactive bridge to facilitate information flow between the action recognizer and the guidance provider (LLM), thereby narrowing the modality gap.
• Figure 2: Illustration of (a) the generation pipeline of class-specific description, and (b) the overall framework of SkeletonAgent.
• Figure 3: Comparisons of previous static descriptions and the targeted distinctions provided by the Questioner. The proposed approach captures key distinctive cues and clearly differentiates them from the features of similar actions.
• Figure 4: Ablation study on the influences of the weights $\alpha$ and $\beta$ under the NTU-60 X-Sub setting with the joint modality.
• Figure 5: Visualization of action descriptions and the topologies learned by GAP xiang2023generative and our method for similar actions Writing and Typing on a Keyboard. Darker colors indicate the stronger correlation between corresponding joints.