CLIP-guided Prototype Modulating for Few-shot Action Recognition

TL;DR

This work presents a CLIP-guided prototype modulating framework called CLIP-FSAR, which consists of two key components: a video-text contrastive objective and a prototype modulation that bridges the task discrepancy between CLIP and the few-shot video task by contrasting videos and corresponding class text descriptions.

Abstract

Learning from large-scale contrastive language-image pre-training like CLIP has shown remarkable success in a wide range of downstream tasks recently, but it is still under-explored on the challenging few-shot action recognition (FSAR) task. In this work, we aim to transfer the powerful multimodal knowledge of CLIP to alleviate the inaccurate prototype estimation issue due to data scarcity, which is a critical problem in low-shot regimes. To this end, we present a CLIP-guided prototype modulating framework called CLIP-FSAR, which consists of two key components: a video-text contrastive objective and a prototype modulation. Specifically, the former bridges the task discrepancy between CLIP and the few-shot video task by contrasting videos and corresponding class text descriptions. The latter leverages the transferable textual concepts from CLIP to adaptively refine visual prototypes with a temporal Transformer. By this means, CLIP-FSAR can take full advantage of the rich semantic priors in CLIP to obtain reliable prototypes and achieve accurate few-shot classification. Extensive experiments on five commonly used benchmarks demonstrate the effectiveness of our proposed method, and CLIP-FSAR significantly outperforms existing state-of-the-art methods under various settings. The source code and models will be publicly available at https://github.com/alibaba-mmai-research/CLIP-FSAR.

Figures (10)

• Figure 1: Performance comparison based on a typical few-shot action recognition method, i.e., OTAM OTAM. We notice that directly replacing the original ImageNet pre-trained backbone with the visual encoder of CLIP CLIP and simply finetuning the CLIP model can only yield limited performance improvements. In contrast, our approach provides significant performance gains by fully exploiting CLIP's multimodal semantic knowledge.
• Figure 2: The architecture of CLIP-FSAR. Given the support and query videos, a visual encoder is first adopted to extract the frame features. Then, we apply a video-text contrastive objective to pull close the obtained video features and the corresponding class text representations. Afterward, a prototype modulation is employed to refine the visual support prototypes with textual semantic features. Finally, a few-shot metric objective is imposed on the resulting informative prototypes and query features for classification.
• Figure 3: Sensitivity analysis of $\alpha$ on SSv2-Small and Kinetics.
• Figure 4: Performance comparison with different numbers of input video frames under the 5-way 1-shot setting on SSv2-Small.
• Figure 5: N-way 1-shot results of our CLIP=FSAction and other baseline methods with N varying from 5 to 10.