DSTSA-GCN: Advancing Skeleton-Based Gesture Recognition with Semantic-Aware Spatio-Temporal Topology Modeling

TL;DR

DSTSA-GCN tackles the challenge of capturing dynamic spatio-temporal topology and multiscale relationships in skeleton-based gesture recognition by introducing three components: GC-GC for channel-specific spatial topology, GT-GC for frame-specific temporal topology, and MS-TCN for multiscale temporal dynamics. A shared Spatio-Temporal Coordinate-Aware (STCA) feature transform enriches representations, and a grouped convolution strategy mitigates local static-topology bias while keeping model complexity in check. Across gesture and action benchmarks, DSTSA-GCN achieves state-of-the-art or competitive results, demonstrating improved sensitivity to temporal variation and distant joint interactions. The approach offers a flexible, interpretable framework with strong practical impact for gesture understanding and human-action analysis in real-world settings.

Abstract

Graph convolutional networks (GCNs) have emerged as a powerful tool for skeleton-based action and gesture recognition, thanks to their ability to model spatial and temporal dependencies in skeleton data. However, existing GCN-based methods face critical limitations: (1) they lack effective spatio-temporal topology modeling that captures dynamic variations in skeletal motion, and (2) they struggle to model multiscale structural relationships beyond local joint connectivity. To address these issues, we propose a novel framework called Dynamic Spatial-Temporal Semantic Awareness Graph Convolutional Network (DSTSA-GCN). DSTSA-GCN introduces three key modules: Group Channel-wise Graph Convolution (GC-GC), Group Temporal-wise Graph Convolution (GT-GC), and Multi-Scale Temporal Convolution (MS-TCN). GC-GC and GT-GC operate in parallel to independently model channel-specific and frame-specific correlations, enabling robust topology learning that accounts for temporal variations. Additionally, both modules employ a grouping strategy to adaptively capture multiscale structural relationships. Complementing this, MS-TCN enhances temporal modeling through group-wise temporal convolutions with diverse receptive fields. Extensive experiments demonstrate that DSTSA-GCN significantly improves the topology modeling capabilities of GCNs, achieving state-of-the-art performance on benchmark datasets for gesture and action recognition, including SHREC17 Track, DHG-14\/28, NTU-RGB+D, and NTU-RGB+D-120.

Figures (6)

• Figure 1: Illustration of the construction of different skeletal topologies. Solid arrows indicate shared topology in the channel or temporal dimension, while dashed lines indicate non-shared. Yellow arrows indicate the direction of the channel and green arrows indicate the direction of the temporal dimension. c and d is based on the dimension-specific of b. a is parameterized in training stage and does not vary in the inference phase with the samples.
• Figure 2: The architecture of Dynamic Spatial-Temporal Semantic Awareness Graph Convolutional Network.
• Figure 3: Visualization of the static topology graphs learned from different initialization strategies. Blue boxes indicate the maximum four distant interactions, and red indicates the maximum four self interactions.
• Figure 4: Visualization comparison of temporal-wise topology graphs (last layer) with CTR-GCN and TD-GCN. Gesture class : Grap. Blue boxes indicate the maximum four distant interactions, and red indicates the maximum four self interactions.
• Figure 5: Class activation mapping results for action sample: Grap, Tap. The horizontal axis represents the joint index and the vertical axis represents the frame (temporal) index.