TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding

TL;DR

TimeViper tackles the challenge of long-video understanding by marrying a state-space based Mamba backbone with Transformer-style attention in a hybrid LLM, and by introducing TransV, an internal token transfer mechanism that compresses vision information into instruction tokens. The authors uncover a vision-to-text aggregation phenomenon and show that vision-token redundancy grows with network depth, enabling aggressive token compression in deeper layers. Through a two-stage training procedure and ToMe-based frame compression, TimeViper processes over 10K frames while maintaining competitive performance on MCQ, TVG, and VDC benchmarks, and it achieves notable efficiency gains in prefilling time and memory. The work advances interpretable, scalable hybrid MLLMs for long-form video understanding and highlights directions for further scaling and data augmentation.

Abstract

We introduce TimeViper, a hybrid vision-language model designed to tackle challenges of long video understanding. Processing long videos demands both an efficient model architecture and an effective mechanism for handling extended temporal contexts. To this end, TimeViper adopts a hybrid Mamba-Transformer backbone that combines the efficiency of state-space models with the expressivity of attention mechanisms. Through this hybrid design, we reveal the vision-to-text information aggregation phenomenon, where information progressively flows from vision tokens to text tokens across increasing LLM depth, resulting in severe vision token redundancy. Motivated by this observation, we propose TransV, a token information transfer module that transfers and compresses vision tokens into instruction tokens while maintaining multimodal understanding capabilities. This design enables TimeViper to process hour-long videos exceeding 10,000 frames. Extensive experiments across multiple benchmarks demonstrate that TimeViper competes with state-of-the-art models while extending frame numbers. We further analyze attention behaviors of both Mamba and Transformer layers, offering new insights into hybrid model interpretability. This work represents an initial step towards developing, interpreting, and compressing hybrid Mamba-Transformer architectures.

Figures (11)

• Figure 1: We present TimeViper, a hybrid Mamba-Transformer vision-language model for efficient long video understanding. We reveal the severe vision token redundancy and a vision-to-text information aggregation phenomenon in hybrid models. To this end, we introduce TransV, the first token-transfer module that compresses vision tokens into text tokens inside the LLM, enabling the model to process over 10,000 frames. Benefitting from the Mamba layers' $O(n)$ computation and $O(1)$ cache cost, TimeViper generates 40.1% more tokens per second than Qwen3 yang2025qwen3 when processing 32k input tokens (approximately 2k frames at 16 tokens per frame) and producing 1k output tokens with batch size 32. TimeViper delivers performance competitive with Transformer-based MLLMs on public benchmarks, including multi-choice QA on VideoMME fu2025videomme (vs. Video-XL shu2025videoxl1), temporal video grounding on Charades sigurdsson2018charades (vs. VTimeLLM huang2024vtimellm), video detailed captioning on VDC chai2025auroracap (vs. AuroraCap chai2025auroracap), and hour-long video understanding on LVBench wang2025lvbench (vs. Gemini-1.5-Pro team2024gemini1.5pro).
• Figure 2: Illustration of TimeViper, our proposed hybrid MLLM for long video understanding. The model consists of a ViT visual encoder, a projector with token merging, and a hybrid Mamba-Transformer LLM equipped with TransV. The token merging bolya2022tokenmerge compresses each frame into 16 vision tokens. Inside the LLM, TransV transfers information from redundant vision tokens to instruction tokens to reduce the number of vision tokens. Specifically, TransV uniformly drops vision tokens in shallow layers and removes low-attention vision tokens in deeper layers. The compression module is implemented through a Gated Cross-Attention mechanism alayrac2022flamingo with adaptive learnable weights. Note that TransV is illustrated before the attention layer for clarity, though it may be applied before any layer in practice.
• Figure 3: Comparison of information blocking to illustrate the vision-to-text information aggregation phenomenon in hybrid MLLMs. For instruction-centric tasks (e.g., multi-choice video QA), information is first aggregated from vision tokens to instruction tokens, which are then used for response generation. In contrast, for vision-centric tasks (e.g., detailed video captioning), vision tokens directly contribute to response generation.
• Figure 4: Illustration of token redundancy. We compare performance under different vision-token dropping rates $p$ using uniform dropping and attention-guided dropping strategies. In the hybrid MLLM, vision token redundancy increases progressively with layer depth, allowing more aggressive token removal in deeper layers with minimal performance loss.
• Figure 5: Comparison of GPU memory usage during inference. While ToMe extends the context window to about 5K frames, TransV efficiently scales beyond 10K frames.