STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding

TL;DR

STOP introduces a dynamic prompting framework that jointly analyzes spatial and temporal cues to adapt frozen CLIP models for video tasks. By generating intra-frame spatial prompts that emphasize discriminative regions and inter-frame temporal prompts that emphasize keyframes with high dynamics, STOP achieves superior action recognition and video-text retrieval performance while remaining parameter-efficient. The method is validated across multiple benchmarks, with ablations showing substantial gains from both prompting components and careful hyper-parameter design. This approach offers a scalable, cost-effective path to leveraging large image-language pretraining for complex video understanding scenarios.

Abstract

Pre-trained on tremendous image-text pairs, vision-language models like CLIP have demonstrated promising zero-shot generalization across numerous image-based tasks. However, extending these capabilities to video tasks remains challenging due to limited labeled video data and high training costs. Recent video prompting methods attempt to adapt CLIP for video tasks by introducing learnable prompts, but they typically rely on a single static prompt for all video sequences, overlooking the diverse temporal dynamics and spatial variations that exist across frames. This limitation significantly hinders the model's ability to capture essential temporal information for effective video understanding. To address this, we propose an integrated Spatial-TempOral dynamic Prompting (STOP) model which consists of two complementary modules, the intra-frame spatial prompting and inter-frame temporal prompting. Our intra-frame spatial prompts are designed to adaptively highlight discriminative regions within each frame by leveraging intra-frame attention and temporal variation, allowing the model to focus on areas with substantial temporal dynamics and capture fine-grained spatial details. Additionally, to highlight the varying importance of frames for video understanding, we further introduce inter-frame temporal prompts, dynamically inserting prompts between frames with high temporal variance as measured by frame similarity. This enables the model to prioritize key frames and enhances its capacity to understand temporal dependencies across sequences. Extensive experiments on various video benchmarks demonstrate that STOP consistently achieves superior performance against state-of-the-art methods. The code is available at https://github.com/zhoujiahuan1991/CVPR2025-STOP.

Figures (7)

• Figure 1: Existing video prompting methods yang2024dglhuang2023vop typically add a static prompt across diverse videos, limiting the model's ability to capture essential temporal information. In contrast, our approach introduces dynamic intra-frame spatial prompts and inter-frame temporal prompts, guiding the model to focus on discriminative regions and key frames with significant temporal dynamics.
• Figure 2: The pipeline of our STOP. For each video, we begin by embedding it into image tokens. Then the intra-frame spatial prompting is introduced to locate discriminative regions and add generated prompts to these areas. Based on the degree of inter-frame variation, we dynamically generate inter-frame prompts and insert them as needed. Finally, these prompts along with the CLS token, pass through the MSA (multi-head self-attention) block to obtain a video representation, which is then used to compute similarity with the text features.
• Figure 3: Attention map visualization results of existing method DGL and STOP (Ours). Visualization of more cases is included in the supplementary material.
• Figure 4: Influence of hyper-parameters of STOP in HDMB51.
• Figure 5: More Attention map visualization results of existing method DGL-Transformer yang2024dgl and STOP (Ours).