Affinity Contrastive Learning for Skeleton-based Human Activity Understanding

TL;DR

ACLNet introduces Affinity Contrastive Learning to skeleton-based human activity understanding by explicitly modeling inter-class semantic affinities via a Motion Family construct and enforcing intra-class margins for hard positives. The inter-class framework uses an affinity similarity derived from direct misclassification signals and indirect contextual overlaps, refined through Motion Family with EMA-updated class centers and a dynamic, family-aware temperature schedule. An intra-class marginal contrastive loss further strengthens separation between hard positives and their nearest negatives, yielding a unified objective: $\mathcal{L} = \mathcal{L}_{ce} + \lambda_{1} \mathcal{L}_{inter} + \lambda_{2} \mathcal{L}_{intra}$. Extensive experiments across NTU-60/120, Kinetics-Skeleton, PKU-MMD, FineGYM, and CASIA-B demonstrate state-of-the-art performance in action recognition, gait, and re-ID, with improved robustness to noisy data and clearer separation of similar actions. The approach offers practical benefits for fine-grained activity analysis and biometric applications by leveraging structured inter-class relationships and controlled intra-class variability.

Abstract

In skeleton-based human activity understanding, existing methods often adopt the contrastive learning paradigm to construct a discriminative feature space. However, many of these approaches fail to exploit the structural inter-class similarities and overlook the impact of anomalous positive samples. In this study, we introduce ACLNet, an Affinity Contrastive Learning Network that explores the intricate clustering relationships among human activity classes to improve feature discrimination. Specifically, we propose an affinity metric to refine similarity measurements, thereby forming activity superclasses that provide more informative contrastive signals. A dynamic temperature schedule is also introduced to adaptively adjust the penalty strength for various superclasses. In addition, we employ a margin-based contrastive strategy to improve the separation of hard positive and negative samples within classes. Extensive experiments on NTU RGB+D 60, NTU RGB+D 120, Kinetics-Skeleton, PKU-MMD, FineGYM, and CASIA-B demonstrate the superiority of our method in skeleton-based action recognition, gait recognition, and person re-identification. The source code is available at https://github.com/firework8/ACLNet.

Figures (6)

• Figure 1: Conceptual diagram for Affinity Contrastive Learning. The neglect of structural commonalities among classes and inherently anomalous positive samples within classes will degrade the performance of existing methods. Therefore, we propose Affinity Contrastive Learning to improve discriminative representations at the inter-class and intra-class levels.
• Figure 2: The framework of the proposed ACLNet. The input skeleton sequence is fed into the GCN backbone to extract skeleton feature $f$, which is embedded into a vector by projection for affinity contrastive learning. Specifically, we introduce affinity similarity to measure the semantic associations between related activities while considering their pairwise and contextual similarities. The Motion Family is then constructed to enable targeted refinement for hard classes. Additionally, we define the affinitive margin to provide accurate control of the minimal distance between the positive sample and the closest negative sample. By increasing the margin, the optimization strategy helps to improve the separation between hard positives and negatives. Finally, the two affinity contrastive losses contribute to the construction of a discriminative feature space, effectively improving the accuracy of the model.
• Figure 3: Examples of Motion Family corresponding to the anchor actions 'reading' and 'wear jacket'. Red and blue areas reflect the notable body parts with high learned weights, which indicate the structural commonality links (e.g., hand-related and arm-related) in the skeleton sequences.
• Figure 4: Ablation study on the effect of different hyper-parameters under the NTU RGB+D 60 X-Sub setting with the joint modality.
• Figure 5: The t-SNE plots of the feature embeddings for five chosen action classes throughout the training process. Colors indicate individual classes from NTU-60 X-Sub. From the early epoch (a) to later epochs (b–d), the clusters grow progressively more compact and more widely separated, revealing a steady gain in class discriminability. Best viewed with zoom in.