MVP-Shot: Multi-Velocity Progressive-Alignment Framework for Few-Shot Action Recognition

TL;DR

This work tackles FSAR by addressing the speed variance of actions through multi-velocity feature representations. It introduces MVP-Shot, which combines Progressive Semantic-Tailored Interaction (PSTI) to inject velocity-specific textual priors into video features and Multi-Velocity Feature Alignment (MVFA) to compute per-velocity similarities and fuse them into a final distance for $N$-way $K$-shot tasks. Using a CLIP-based visual encoder and OTAM-style temporal metrics, MVP-Shot achieves state-of-the-art results on HMDB51, UCF101, Kinetics, and SSv2-small, with notable gains in the low-shot regime. The approach demonstrates the value of multi-velocity matching for robust cross-speed action recognition and offers a practical path toward more flexible few-shot video understanding.

Abstract

Recent few-shot action recognition (FSAR) methods typically perform semantic matching on learned discriminative features to achieve promising performance. However, most FSAR methods focus on single-scale (e.g., frame-level, segment-level, etc) feature alignment, which ignores that human actions with the same semantic may appear at different velocities. To this end, we develop a novel Multi-Velocity Progressive-alignment (MVP-Shot) framework to progressively learn and align semantic-related action features at multi-velocity levels. Concretely, a Multi-Velocity Feature Alignment (MVFA) module is designed to measure the similarity between features from support and query videos with different velocity scales and then merge all similarity scores in a residual fashion. To avoid the multiple velocity features deviating from the underlying motion semantic, our proposed Progressive Semantic-Tailored Interaction (PSTI) module injects velocity-tailored text information into the video feature via feature interaction on channel and temporal domains at different velocities. The above two modules compensate for each other to make more accurate query sample predictions under the few-shot settings. Experimental results show our method outperforms current state-of-the-art methods on multiple standard few-shot benchmarks (i.e., HMDB51, UCF101, Kinetics, and SSv2-small).

Figures (7)

• Figure 1: The main idea of this work. (a): The previous methods focus on single-velocity feature alignment, such as frame-level and segment-level. (b): In our work, we capture multi-velocity feature representation and perform multi-velocity feature alignment to deal with action instances with diverse speeds.
• Figure 2: The overview of the proposed Multi-Velocity Progressive-Alignment (MVP-shot) Framework. The support and query videos are first fed into a visual encoder to extract frame-level features. Then, based on frame features, we apply a Progressive Semantic-Tailored Interaction (PSTI) module to capture multi-velocity action features. Accordingly, we employ a Multi-Velocity Feature Alignment (MVFA) module to measure pair-wise semantic similarity at different velocity scales. For clarity, the figure does not depict other support videos in the few-shot task.
• Figure 3: Illustration of Semantic-Tailored Interaction (STI) module. STI consists of Temporal Relation Network and Channel Correction Network, which perform feature interaction in the temporal and channel domains, respectively.
• Figure 4: $N$-way $1$-shot results of our MVP-shot on Kinetics carreira2017quo dataset, our baseline methods and CLIP-FSAR with $N$ varying from $5$ to $10$.
• Figure 5: Comparison results (§\ref{['ablatio']}) with different numbers of support samples on kinetics carreira2017quo dataset under $5$-way $K$-shot settings.