CoCoDiff: Diversifying Skeleton Action Features via Coarse-Fine Text-Co-Guided Latent Diffusion

TL;DR

CoCoDiff tackles limited feature diversity and semantic drift in skeleton-based action recognition by generating diverse yet semantically grounded latent features via a latent diffusion model. It conditions diffusion with coarse (labels and synonyms) and fine (detailed action descriptions) text guided by GPT-3.5 and a text encoder, while using a GCN to extract high-level skeleton features. The method employs a two-stage training regime and a skeleton-text contrastive loss to maintain semantic alignment, achieving state-of-the-art results on NTU RGB+D, NTU RGB+D 120, and Kinetics-Skeleton without increasing inference cost. Practically, CoCoDiff provides a plug-and-play enhancement to existing backbones, improving generalization on large intra-class variation datasets with efficient training.”

Abstract

In action recognition tasks, feature diversity is essential for enhancing model generalization and performance. Existing methods typically promote feature diversity by expanding the training data in the sample space, which often leads to inefficiencies and semantic inconsistencies. To overcome these problems, we propose a novel Coarse-fine text co-guidance Diffusion model (CoCoDiff). CoCoDiff generates diverse yet semantically consistent features in the latent space by leveraging diffusion and multi-granularity textual guidance. Specifically, our approach feeds spatio-temporal features extracted from skeleton sequences into a latent diffusion model to generate diverse action representations. Meanwhile, we introduce a coarse-fine text co-guided strategy that leverages textual information from large language models (LLMs) to ensure semantic consistency between the generated features and the original inputs. It is noted that CoCoDiff operates as a plug-and-play auxiliary module during training, incurring no additional inference cost. Extensive experiments demonstrate that CoCoDiff achieves SOTA performance on skeleton-based action recognition benchmarks, including NTU RGB+D, NTU RGB+D 120 and Kinetics-Skeleton.

Figures (6)

• Figure 1: Comparison of the proposed Coarse-fine text co-guidance Diffusion model (CoCoDiff) framework with other traditional methods. The different colored geometric shapes in this figure represent action features of different categories.
• Figure 2: Overall framework of the Coarse-fine text Co-guidance Diffusion model (CoCoDiff). During the training phase, high-level skeleton action features are extracted using a feature extraction network and served as inputs to the latent diffusion model for generating diverse action features. Then, the action description as fine text are incorporated as a condition in each iteration of denoising process. And action labels as coarse text embedding is employed to constrain their semantic consistency with generated skeleton features $\bm{{\widehat{x}_0}}$ using contrastive loss. In the inference phase, a well-trained feature extraction network is employed for action classification.
• Figure 3: Coarse-fine text co-guided strategy for the action "throw". Two types of textual descriptions for each action are generated using a large language model (GPT-3.5) and using text encoder for feature embedding. Subsequently, the fine text feature is employed to progressively guide the generation of features during the diffusion process. And coarse text feature is utilized for contrastive learning with the generated action features.
• Figure 4: Coarse and fine text generated from different prompt inputs by GPT-3.5.
• Figure 5: The group-wise accuracy difference ($\%$) between our method and HD-GCN on large intra-class variations actions for NTU RGB+D 120 dataset under the X-Sub setting with the joint input modality.