BIMBA: Selective-Scan Compression for Long-Range Video Question Answering

TL;DR

BIMBA addresses the challenge of long-form video question answering by transforming massive sequences of spatiotemporal tokens into a compact, information-rich representation via a selective-scan state-space token selector. It introduces interleaved queries, bidirectional selective scanning, and optional question-conditioned token selection to capture long-range dependencies efficiently, enabling LLM-based reasoning with linear-scale computation. Across seven long-form VQA benchmarks, BIMBA achieves state-of-the-art accuracy and exhibits favorable memory and runtime characteristics compared with self-attention and pooling baselines, including strong performance when processing hundreds of thousands of tokens. The approach significantly advances practical, scalable multimodal reasoning for long videos and opens avenues for tasks like video summarization and hierarchical video modeling.

Abstract

Video Question Answering (VQA) in long videos poses the key challenge of extracting relevant information and modeling long-range dependencies from many redundant frames. The self-attention mechanism provides a general solution for sequence modeling, but it has a prohibitive cost when applied to a massive number of spatiotemporal tokens in long videos. Most prior methods rely on compression strategies to lower the computational cost, such as reducing the input length via sparse frame sampling or compressing the output sequence passed to the large language model (LLM) via space-time pooling. However, these naive approaches over-represent redundant information and often miss salient events or fast-occurring space-time patterns. In this work, we introduce BIMBA, an efficient state-space model to handle long-form videos. Our model leverages the selective scan algorithm to learn to effectively select critical information from high-dimensional video and transform it into a reduced token sequence for efficient LLM processing. Extensive experiments demonstrate that BIMBA achieves state-of-the-art accuracy on multiple long-form VQA benchmarks, including PerceptionTest, NExT-QA, EgoSchema, VNBench, LongVideoBench, and Video-MME. Code, and models are publicly available at https://sites.google.com/view/bimba-mllm.

Figures (11)

• Figure 1: Our proposed BIMBA model uses a Mamba-based Spatiotemporal Token Selector to select a reduced number of salient tokens from a long sequence of features extracted via a pretrained image encoder. The token selection is optionally conditioned using the textual query to identify the features that are most informative for answering a given question. Finally, the selected and transformed tokens are passed to a large language model with a tokenized version of the input question to generate the answer.
• Figure 2: (a): Architecture of our Spatiotemporal Token Selector. (b): Traditional selective scan with queries appended at the sequence's start or end introduces positional biases that often lead to suboptimal performance. (c) We propose to interleave the queries uniformly to capture interactions between spatiotemporal tokens across the video more evenly. (d) Furthermore, we introduce a bidirectional selective scan (forward and backward) operation to improve the long-range modeling further.
• Figure 3: Accuracy achieved by BIMBA and baseline models on NeXT-QA (left) and EgoSchema (right) as a function of the number of input tokens for models based on LLaVA (top row) and LLaMA (bottom row). BIMBA achieves the highest accuracy for all sequence lengths, and the difference with other baselines increases as we increase the number of input tokens. Self-attention cannot be applied to long sequences as it causes GPU out-of-memory issues once the number of tokens becomes too large.
• Figure 4: Computation cost of BIMBA and baseline models in terms of memory usage (left) and runtime (right). All models are based on LLaVA. Models based on self-attention or that do not perform compression (Vanilla) run quickly out of memory as the number of input tokens is increased. The runtime of BIMBA grows gracefully as a function of the input sequence length, unlike for the case of Vanilla.
• Figure S1: Relative performance improvement of (left) BIMBA-LLaVA over PLLaVA baseline and (right) BIMBA-LLaMA over LLaMA-3.2 (video) baseline for different video durations on NextQA dataset. Our model achieves larger gains as the video length increases.