State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding

TL;DR

The paper tackles efficient downstream adaptation of pre-trained state-space video models by introducing State Space Prompting (SSP), which combines Intra-Frame Gathering (IFG) and Inter-Frame Spreading (IFS) to gather local spatial cues and spread global temporal context. SSP is designed for the VideoMamba backbone, and it achieves state-of-the-art results on multiple video benchmarks while tuning only a small fraction of parameters (~3%). The method demonstrates strong improvements on long videos by reducing information transmission paths and effectively propagating discriminative spatio-temporal features. This approach offers practical benefits for deploying video foundation models on resource-constrained settings without sacrificing performance.

Abstract

Recently, pre-trained state space models have shown great potential for video classification, which sequentially compresses visual tokens in videos with linear complexity, thereby improving the processing efficiency of video data while maintaining high performance. To apply powerful pre-trained models to downstream tasks, prompt learning is proposed to achieve efficient downstream task adaptation with only a small number of fine-tuned parameters. However, the sequentially compressed visual prompt tokens fail to capture the spatial and temporal contextual information in the video, thus limiting the effective propagation of spatial information within a video frame and temporal information between frames in the state compression model and the extraction of discriminative information. To tackle the above issue, we proposed a State Space Prompting (SSP) method for video understanding, which combines intra-frame and inter-frame prompts to aggregate and propagate key spatiotemporal information in the video. Specifically, an Intra-Frame Gathering (IFG) module is designed to aggregate spatial key information within each frame. Besides, an Inter-Frame Spreading (IFS) module is designed to spread discriminative spatio-temporal information across different frames. By adaptively balancing and compressing key spatio-temporal information within and between frames, our SSP effectively propagates discriminative information in videos in a complementary manner. Extensive experiments on four video benchmark datasets verify that our SSP significantly outperforms existing SOTA methods by 2.76% on average while reducing the overhead of fine-tuning parameters.

Figures (12)

• Figure 1: Existing video prompting methods on Mamba and its visualization results of updating date compared to our SSP. Existing methods directly concatenate learnable prompts to video tokens, resulting in the information decay problem after the long-term state space compression. However, our SSP achieves information enhancement through spatial gathering and temporal spreading.
• Figure 2: The pipeline of our SSP. We embed videos into image tokens with spatial prompts. After the initial Mamba layer, our complementary IFG and IFS modules operate - IFG aggregates spatial information while IFS spreads temporal information. Information entropy and frame-specific spatial variance bridge these modules. The prompted tokens and CLS token then pass through subsequent Mamba layers for video classification.
• Figure 3: Visualization of our intra-frame prompts, which capture discriminative local features.
• Figure 4: Visualization of our inter-frame prompts. The action "catch" can be parsed into two distinct phases: preparation for the catch and the actual reception of the ball. Our inter-frame prompts effectively locate the key transitional moments between these critical phases.
• Figure 5: Comparison on Breakfast. SSP outperforms existing methods while tuning minimal parameters.