Variational Contrastive Learning for Skeleton-based Action Recognition

TL;DR

This paper tackles skeleton-based action recognition under limited supervision by unifying contrastive self-supervised learning with variational latent modeling. It introduces a variational contrastive learning framework that adds Gaussian latent sampling to a MoCo-like architecture, yielding a unified objective $\mathcal{L}_{\text{VCL}}(x) = \ell_{\text{InfoNCE}}(x) + D_{\text{KL}}(q_{\phi}(z|x) \| p(z))$, and it employs a three-stream fusion (joint, bone, motion) to capture comprehensive motion cues. Empirical results on NTU-60, NTU-120, and PKU-MMD demonstrate strong performance, particularly in low-label settings, with ablations highlighting the benefits of variational framing and motion-focused representations. Qualitative analyses corroborate that the learned embeddings emphasize semantically relevant joints and dynamics, suggesting improved robustness and transferability across datasets and supervision levels.

Abstract

In recent years, self-supervised representation learning for skeleton-based action recognition has advanced with the development of contrastive learning methods. However, most of contrastive paradigms are inherently discriminative and often struggle to capture the variability and uncertainty intrinsic to human motion. To address this issue, we propose a variational contrastive learning framework that integrates probabilistic latent modeling with contrastive self-supervised learning. This formulation enables the learning of structured and semantically meaningful representations that generalize across different datasets and supervision levels. Extensive experiments on three widely used skeleton-based action recognition benchmarks show that our proposed method consistently outperforms existing approaches, particularly in low-label regimes. Moreover, qualitative analyses show that the features provided by our method are more relevant given the motion and sample characteristics, with more focus on important skeleton joints, when compared to the other methods.

Figures (4)

• Figure 1: Overview of the proposed framework which includes variational projection heads that enable Gaussian sampling for contrastive pretraining.
• Figure 2: Qualitative comparison of the embedding distribution between SkeletonCLR and the proposed method under different supervision settings.
• Figure 3: Grad-CAM visualizations of joint importance over time for skeleton-based action recognition, shown at 10, 25, 35, 45, and 50 time steps (left to right), of SkeletonCLR and our proposed method. Left column corresponds to the "Eating Meal" action under linear evaluation, while the right column shows "Taking off Jacket" action under 1% labeled semi-supervised training. Lighter-colored joints indicate higher relevance.
• Figure 4: Left: UMAP visualizations of learned representations on PKU-MMD Part I under different supervision levels, with CrosSCLR (top row) and ours (bottom row). Right: Methods focus shown at frames 10, 25, 35, 45, and 50 - for the 1% labeled setting.