Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIP

TL;DR

The paper tackles zero-shot action recognition by requiring collaborative multi-modal spatiotemporal understanding of both static and dynamic action cues. It introduces Spatiotemporal Dynamic Duo (STDD), which augments CLIP with Space-Time Cross Attention to capture cross-frame dynamics and an Action Semantic Knowledge Graph–driven spatiotemporal text augmentation to generate nuanced prompts. Training aligns frame-level video representations with refined text prompts while leveraging distillation from a frozen CLIP to preserve generalization. Across Kinetics-600, UCF101, and HMDB51, STDD achieves state-of-the-art performance under ZSAR settings, validating effective temporal modeling without additional parameters and highlighting the value of structured semantic prompts for video-text alignment.

Abstract

Zero-shot action recognition (ZSAR) requires collaborative multi-modal spatiotemporal understanding. However, finetuning CLIP directly for ZSAR yields suboptimal performance, given its inherent constraints in capturing essential temporal dynamics from both vision and text perspectives, especially when encountering novel actions with fine-grained spatiotemporal discrepancies. In this work, we propose Spatiotemporal Dynamic Duo (STDD), a novel CLIP-based framework to comprehend multi-modal spatiotemporal dynamics synergistically. For the vision side, we propose an efficient Space-time Cross Attention, which captures spatiotemporal dynamics flexibly with simple yet effective operations applied before and after spatial attention, without adding additional parameters or increasing computational complexity. For the semantic side, we conduct spatiotemporal text augmentation by comprehensively constructing an Action Semantic Knowledge Graph (ASKG) to derive nuanced text prompts. The ASKG elaborates on static and dynamic concepts and their interrelations, based on the idea of decomposing actions into spatial appearances and temporal motions. During the training phase, the frame-level video representations are meticulously aligned with prompt-level nuanced text representations, which are concurrently regulated by the video representations from the frozen CLIP to enhance generalizability. Extensive experiments validate the effectiveness of our approach, which consistently surpasses state-of-the-art approaches on popular video benchmarks (i.e., Kinetics-600, UCF101, and HMDB51) under challenging ZSAR settings.

Figures (8)

• Figure 1: Illustration of the challenges without collaborative multi-modal spatiotemporal understanding. (a) A model lacking static context alignment may misidentify the novel class due to the ambiguity associated with the barbell. (b) It might also struggle generalizing to other novel weightlifting actions, due to the subtle dynamic differences and strong visual similarities.
• Figure 2: Overview of our framework. With a four-step operation applied within each block, we transform the spatial attention into novel Space-time Cross Attention. Spatiotemporal text augmentation is conducted to derive spatial and temporal text prompts, where multi-modal dynamics are meticulously aligned in a fine-grained manner.
• Figure 3: Illustration of our method. (1) We extend the spatial attention block to perform Space-time Cross Attention by applying Window Shift Masking to the input spatial tokens, and perform Multi-Scale Channel Mixing to capture temporal dynamics before MHSA. Then, we employ the spatial padding strategy to fill in the masked positions for seamless short-cutting, fusing additional dynamics effortlessly. (2) We conduct spatiotemporal text augmentation to obtain nuanced spatial and temporal text prompts by elaborating on static and dynamic concepts with their interrelations presented in ASKG.
• Figure 4: Effect of different combinations of text augmentation and alignment mechanisms for our method and CLIP.
• Figure 5: Visualizations of the attention maps and frame-prompt alignment scores of Archery. Our framework consistently prioritizes local body parts and objects participated in the dynamic movements.