LLM Enhanced Action Recognition via Hierarchical Global-Local Skeleton-Language Model

Abstract

Skeleton-based human action recognition has achieved remarkable progress in recent years. However, most existing GCN-based methods rely on short-range motion topologies, which not only struggle to capture long-range joint dependencies and complex temporal dynamics but also limit cross-modal semantic alignment and understanding due to insufficient modeling of action semantics. To address these challenges, we propose a hierarchical global-local skeleton-language model (HocSLM), enabling the large action model be more representative of action semantics. First, we design a hierarchical global-local network (HGLNet) that consists of a composite-topology spatial module and a dual-path hierarchical temporal module. By synergistically integrating multi-level global and local modules, HGLNet achieves dynamically collaborative modeling at both global and local scales while preserving prior knowledge of human physical structure, significantly enhancing the model's representation of complex spatio-temporal relationships. Then, a large vision-language model (VLM) is employed to generate textual descriptions by passing the original RGB video sequences to this model, providing the rich action semantics for further training the skeleton-language model. Furthermore, we introduce a skeleton-language sequential fusion module by combining the features from HGLNet and the generated descriptions, which utilizes a skeleton-language model (SLM) for aligning skeletal spatio-temporal features and textual action descriptions precisely within a unified semantic space. The SLM model could significantly enhance the HGLNet's semantic discrimination capabilities and cross-modal understanding abilities. Extensive experiments demonstrate that the proposed HocSLM achieves the state-of-the-art performance on three mainstream benchmark datasets: NTU RGB+D 60, NTU RGB+D 120, and Northwestern-UCLA.

Figures (7)

• Figure 1: The architecture of the proposed HocSLM, including 1) Skeleton Component, 2) Language Component, and 3) Fusion Component.
• Figure 2: Overall architecture of the CTS. The leftmost module is the feature preprocessing block. The upper orange region corresponds to the GSM, and the lower green region corresponds to the LSE.
• Figure 3: Overall architecture of the DHT. The yellow region on the left half corresponds to the GTA, and the purple region on the right half corresponds to the LTC.
• Figure 4: The overall architecture of SSF.
• Figure 5: The overall architecture of 6-stream ensemble.