Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition

TL;DR

DiST, an innovative Decomposition-incorporation framework for FSAR that makes use of decoupled Spatial and Temporal knowledge provided by large language models to learn expressive multi-granularity prototypes, is proposed.

Abstract

Few-Shot Action Recognition (FSAR) is a challenging task that requires recognizing novel action categories with a few labeled videos. Recent works typically apply semantically coarse category names as auxiliary contexts to guide the learning of discriminative visual features. However, such context provided by the action names is too limited to provide sufficient background knowledge for capturing novel spatial and temporal concepts in actions. In this paper, we propose DiST, an innovative Decomposition-incorporation framework for FSAR that makes use of decoupled Spatial and Temporal knowledge provided by large language models to learn expressive multi-granularity prototypes. In the decomposition stage, we decouple vanilla action names into diverse spatio-temporal attribute descriptions (action-related knowledge). Such commonsense knowledge complements semantic contexts from spatial and temporal perspectives. In the incorporation stage, we propose Spatial/Temporal Knowledge Compensators (SKC/TKC) to discover discriminative object-level and frame-level prototypes, respectively. In SKC, object-level prototypes adaptively aggregate important patch tokens under the guidance of spatial knowledge. Moreover, in TKC, frame-level prototypes utilize temporal attributes to assist in inter-frame temporal relation modeling. These learned prototypes thus provide transparency in capturing fine-grained spatial details and diverse temporal patterns. Experimental results show DiST achieves state-of-the-art results on five standard FSAR datasets.

Figures (6)

• Figure 1: Our main idea. DiST decomposes category names into diverse spatio-temporal knowledge, and makes use of such decoupled knowledge to guide the learning of object-level and frame-level prototypes, respectively.
• Figure 2: Overview of DiST. Video inputs are first processed by the visual encoder of CLIP to obtain initial patch-level and frame-level features. Then, we leverage LLM to decompose vanilla action names into action-related background knowledge (§\ref{['generation']}). Furthermore, SKC/TKC incorporate decoupled prior knowledge and visual features to form discriminative object- and frame-level prototypes for spatial/temporal matching (§\ref{['SKC']} and §\ref{['TKC']}). Finally, we can combine spatial and temporal matching results to obtain the merged query prediction (§\ref{['metric']}).
• Figure 3: Illustration of Spatial Knowledge Compensator (SKC) (§\ref{['SKC']}). SKC aims to learn discriminative object-level prototypes in a sparse aggregation manner via patch aggregation and attribute injection.
• Figure 4: Left: The impact of the varying fusion parameter$\alpha$ on HMDB51 kuehne2011hmdb in the $5$-way $1$-shot setting (see §\ref{['sec44']}). Right: 5-way 1-shot class improvement of DiST compared to CLIP-FSAR wang2023clip on all class action classes on HMDB51 kuehne2011hmdb (see §\ref{['sec45']}). Our DiST achieves improvement on all action classes.
• Figure 5: Visualization of spatial and temporal prompts under the $1$-shot setting (§\ref{['sec45']}). The spatial prompts are shown as highlighted response areas in each frame. We also show cross-attention temporal prompt weights of Eq. \ref{['gongshi']} in a line graph.