Multi-view Distillation based on Multi-modal Fusion for Few-shot Action Recognition(CLIP-$\mathrm{M^2}$DF)

TL;DR

This work tackles few-shot action recognition by mitigating distribution overlap and outlier sensitivity through a multi-modal, multi-view framework grounded in CLIP. A Probability Prompt Selector leverages prompt-text consistency to augment query information, while Local and Global Temporal Context Extractors feed into a Cross Transformer-based Multi-modal Fusion Encoder. The proposed Mutual Distillation between two views enforces cross-view agreement, enhancing robustness to distribution bias. Extensive experiments on UCF101, HMDB51, Kinetics, and SSv2 demonstrate state-of-the-art performance and strong class-wise improvements, underscoring the practical value of integrating prompts, temporal context, and cross-view distillation for few-shot action recognition.

Abstract

In recent years, few-shot action recognition has attracted increasing attention. It generally adopts the paradigm of meta-learning. In this field, overcoming the overlapping distribution of classes and outliers is still a challenging problem based on limited samples. We believe the combination of Multi-modal and Multi-view can improve this issue depending on information complementarity. Therefore, we propose a method of Multi-view Distillation based on Multi-modal Fusion. Firstly, a Probability Prompt Selector for the query is constructed to generate probability prompt embedding based on the comparison score between the prompt embeddings of the support and the visual embedding of the query. Secondly, we establish a Multi-view. In each view, we fuse the prompt embedding as consistent information with visual and the global or local temporal context to overcome the overlapping distribution of classes and outliers. Thirdly, we perform the distance fusion for the Multi-view and the mutual distillation of matching ability from one to another, enabling the model to be more robust to the distribution bias. Our code is available at the URL: \url{https://github.com/cofly2014/MDMF}.

Figures (11)

• Figure 1: The framework. The grey regions are the modules or some simple operations. The blue dashed regions represent the features. The khaki arrows are the data flow of support, including the visual and label. The yellow arrows are the data flow of the query, including the visual and label. There are 6 modules in our model. (1) CLIP(Visual Encoder and Text Encoder). (2) Probability Prompt Selector (PPS). (3) Local Temporal Context Extractor (LTCE). (4) Global Temporal Context Extractor (GTCE). (5) Multi-modal Fusion Encoder (MMFE). (6) Multiple-view Mutual Distillation(MVMD). There are some other operations: (a) Distillation Condition. (b) Comparison. (c) Main loss
• Figure 2: The left part is the LTCE. It contains several Conv1d operations. According to Conv1d, the features focus on the local temporal context. The middle part is the GTCE. The core part is TCN. We add the last frame of the output to each input frame. The features focus on the global temporal context. The right part is the Cross Transformer. Features that are from the LTCE (GTCE) and features that are directly from the CLIP visual encoder are concatenated with the prompt embedding. The features from LTCE (GTCE) are used as the Queries, and the features from the CLIP visual encoder are used as the Keys and Values.
• Figure 3: The distillation condition. The up is the local temporal context view for visual comparison and text comparison. The down is the global temporal context view for visual comparison and text comparison. We can see the $0.7633>0.6299$ and $0.24>0.2254$, so the local temporal context view is reliable.
• Figure 4: Hyper-parameter $\lambda$ for accuracy.
• Figure 5: Distribution Comparison on Kinetics.