Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers

TL;DR

An orthogonal direction is explored to build a hybrid Mamba-Transformer model (VAMBA) that employs Mamba-2 blocks to encode video tokens with linear complexity that achieves at least 50% reduction in GPU memory usage during training and inference, and nearly doubles the speed per training step compared to transformer-based LMMs.

Abstract

State-of-the-art transformer-based large multimodal models (LMMs) struggle to handle hour-long video inputs due to the quadratic complexity of the causal self-attention operations, leading to high computational costs during training and inference. Existing token compression-based methods reduce the number of video tokens but often incur information loss and remain inefficient for extremely long sequences. In this paper, we explore an orthogonal direction to build a hybrid Mamba-Transformer model (VAMBA) that employs Mamba-2 blocks to encode video tokens with linear complexity. Without any token reduction, VAMBA can encode more than 1024 frames (640$\times$360) on a single GPU, while transformer-based models can only encode 256 frames. On long video input, VAMBA achieves at least 50% reduction in GPU memory usage during training and inference, and nearly doubles the speed per training step compared to transformer-based LMMs. Our experimental results demonstrate that VAMBA improves accuracy by 4.3% on the challenging hour-long video understanding benchmark LVBench over prior efficient video LMMs, and maintains strong performance on a broad spectrum of long and short video understanding tasks.

Figures (6)

• Figure 1: Vamba achieves strong long video understanding performance (42.1% on LVBench wang2024lvbench) while being more computationally efficient compared to transformer-based LMMs.
• Figure 2: Overview of our Vamba model architecture. Compared to transformer-based LMMs (left), we replace the costly causal self-attention operations with the more efficient cross-attention layers and Mamba blocks to achieve better efficiency.
• Figure 3: Comparison of training GPU memory usage and runtime per training step between Qwen2-VL-7B and Vamba.
• Figure 4: Comparison of GPU memory usage and FLOPs between Qwen2-VL-7B and Vamba during inference.
• Figure 5: Qualitative comparison between Vamba and efficient LMM baselines. Red text denotes incorrect responses, while green text highlights the correct responses by our Vamba.