VideoMamba: Spatio-Temporal Selective State Space Model

TL;DR

VideoMamba extends the pure Mamba selective State Space Model to video understanding, achieving competitive accuracy with linear computational complexity by processing spatio-temporal tokens via bidirectional ST-SSMs. It introduces a video tokenizer, learnable positional embeddings, and a stack of ST-SSM encoder blocks to capture non-sequential spatial information alongside sequential temporal dynamics. Across HMDB51, Something-Something V2, and Kinetics-400, VideoMamba delivers strong results with substantially lower GFLOPs and memory usage than transformer-based rivals, while enabling robust long-range video modeling. Additional analyses demonstrate the importance of temporal consistency, Delta-based context gating, and pretraining, and the approach extends effectively to long-term video tasks and other video understanding problems such as action detection and temporal segmentation.

Abstract

We introduce VideoMamba, a novel adaptation of the pure Mamba architecture, specifically designed for video recognition. Unlike transformers that rely on self-attention mechanisms leading to high computational costs by quadratic complexity, VideoMamba leverages Mamba's linear complexity and selective SSM mechanism for more efficient processing. The proposed Spatio-Temporal Forward and Backward SSM allows the model to effectively capture the complex relationship between non-sequential spatial and sequential temporal information in video. Consequently, VideoMamba is not only resource-efficient but also effective in capturing long-range dependency in videos, demonstrated by competitive performance and outstanding efficiency on a variety of video understanding benchmarks. Our work highlights the potential of VideoMamba as a powerful tool for video understanding, offering a simple yet effective baseline for future research in video analysis.

Figures (12)

• Figure 1: Performance and efficiency comparisons among ImageNet-1K pretrained video models. VideoMamba shows superior or comparable performance to VideoSwin-T liu2022video, while having clear advantage in terms of reduced GFLOPs and memory consumption compared to VideoSwin-T liu2022video and VideoMAE-S wang2023videomae.
• Figure 1: Experimental result of VideoMamba's dependency on temporal consistency. We report Top-1 accuracy of ImageNet pretrained model on HMDB dataset.
• Figure 2: Comprehensive View of VideoMamba's Framework. (a) Architecture of VideoMamba. This includes the initial tokenization of video frames, addition of positional embeddings, and processing through encoder blocks that utilize proposed Spatio-Temporal Forward and Backward SSMs for thorough video analysis. (b) Process of Spatio-Temporal Forward and Backward Scanning within the SSMs, with white dashed arrows indicating the scanning direction of video tokens.
• Figure 3: Details of proposed Spatio-Temporal SSM. The figure illustrates the external and internal operations of the Spatio-Temporal Forward & Backward SSM. Here, the backward scanning method represents Spatio-temporal reversal.
• Figure 3: Positional embedding and initialization method from ImageNet pretrained model on SSV2 and HMDB, as stated in Section \ref{['section:embedding']}.