VideoMAP: Toward Scalable Mamba-based Video Autoregressive Pretraining

TL;DR

VideoMAP tackles scalability and sample efficiency in video pretraining by combining a hybrid Mamba-Transformer backbone with a frame-wise autoregressive objective. A 4-to-1 Mamba-Transformer ratio preserves efficiency while expanding capacity, and frame-wise decoding captures temporal dynamics while aligning with CLIP targets. Empirically, VideoMAP yields strong results on Kinetics-400, Something-Something V2, Breakfast, COIN, and enables memory-efficient use as a visual encoder for VideoLLMs, with notable data efficiency on smaller datasets. The work provides open-source code and demonstrates potential for broader multimodal integration.

Abstract

Recent Mamba-based architectures for video understanding demonstrate promising computational efficiency and competitive performance, yet struggle with overfitting issues that hinder their scalability. To overcome this challenge, we introduce VideoMAP, a Hybrid Mamba-Transformer framework featuring a novel pre-training approach. VideoMAP uses a 4:1 Mamba-to-Transformer ratio, effectively balancing computational cost and model capacity. This architecture, combined with our proposed frame-wise masked autoregressive pre-training strategy, delivers significant performance gains when scaling to larger models. Additionally, VideoMAP exhibits impressive sample efficiency, significantly outperforming existing methods with less training data. Experiments show that VideoMAP outperforms existing models across various datasets, including Kinetics-400, Something-Something V2, Breakfast, and COIN. Furthermore, we demonstrate the potential of VideoMAP as a visual encoder for multimodal large language models, highlighting its ability to reduce memory usage and enable the processing of longer video sequences. The code is open-source at https://github.com/yunzeliu/MAP

Figures (6)

• Figure 1: We propose VideoMAP for scalable Mamba-based video autoregressive pretraining. We propose to use a hybrid Mamba-Transformer network and a frame-wise autoregressive prediction method demonstrating stronger scalability and sample efficiency. Unlike the UMTli2023unmasked used in VideoMambali2025videomamba, our proposed VideoMAP requires the network to predict both the spatial and semantic information of future frames simultaneously.
• Figure 2: We propose Video Masked Autoregressive Pretraining (VideoMAP) to pretrain the hybrid Mamba-Transformer vision backbones. This strategy combines the strengths of both UMT and frame-level autoregressive decoder, improving the performance of Transformer and Mamba modules within a unified paradigm.
• Figure 3: We propose a 4:1 ratio Mamba-Transformer block.
• Figure 4: We propose two autoregressive decoding strategies: frame by frame and token by token.
• Figure 5: We propose a VideoLLM based on VideoMAP and MAP and significantly reduce the amount of GPU memory.