A Generically Contrastive Spatiotemporal Representation Enhancement for 3D Skeleton Action Recognition

TL;DR

This work tackles ambiguous samples in skeleton-based action recognition by explicitly modeling latent data distributions. It introduces CSRE, a generically plug-and-play framework that decomposes spatiotemporal representations into spatial-specific and temporal-specific features using STFD and applies cross-sequence contrast with dual memory banks via two InfoNCE losses, combined with standard cross-entropy. The total objective is $L = L_{CE} + L_{NCE}^{spa} + L_{NCE}^{tem}$. Empirically, CSRE shows consistent improvements across five representative encoders on five benchmarks, demonstrating strong generalization and potential for robust, fine-grained action recognition.

Abstract

Skeleton-based action recognition is a central task in computer vision and human-robot interaction. However, most previous methods suffer from overlooking the explicit exploitation of the latent data distributions (i.e., the intra-class variations and inter-class relations), thereby leading to confusion about ambiguous samples and sub-optimum solutions of the skeleton encoders. To mitigate this, we propose a Contrastive Spatiotemporal Representation Enhancement (CSRE) framework to obtain more discriminative representations from the sequences, which can be incorporated into various previous skeleton encoders and can be removed when testing. Specifically, we decompose the representation into spatial- and temporal-specific features to explore fine-grained motion patterns along the corresponding dimensions. Furthermore, to explicitly exploit the latent data distributions, we employ the attentive features to contrastive learning, which models the cross-sequence semantic relations by pulling together the features from the positive pairs and pushing away the negative pairs. Extensive experiments show that CSRE with five various skeleton encoders (HCN, 2S-AGCN, CTR-GCN, Hyperformer, and BlockGCN) achieves solid improvements on five benchmarks. The code will be released at https://github.com/zhshj0110/CSRE.

Figures (6)

• Figure 1: Comparison between previous method (in green) and proposed CSRE framework (in blue). Compared to the previous method, our CSRE can enable the skeleton encoder better to explore the fine-grained motion patterns along the spatial and temporal dimensions, which is beneficial for recognizing ambiguous action samples.
• Figure 2: Visualization of samples from two confused classes ("take off glasses" (blue) and "take off a hat" (red). (a) Visualization of sequences performed by different persons (each row represents a different person) in the same view. (b) Visualization of feature representation by t-SNE for the corresponding action in the test set. The left is from the CTR-GCN, while the right is from our method. Each color denotes a certain class.
• Figure 3: The architecture of the STFD module.
• Figure 4: Framework of the proposed CSRE. The black dotted lines represent the training process, and the red solid line represents the testing process. The skeleton encoder extracts spatial-temporal representation $\boldsymbol{X}$ from the input skeleton sequence $\boldsymbol{S}$. Then,
• Figure 5: Action class with accuracy differences higher than 3% between CTR-GCN and our method in the X-sub benchmark of NTU120 dataset.