Test-Time Temporal Sampling for Efficient MLLM Video Understanding

TL;DR

The paper tackles the challenge of long-video understanding with multimodal large language models by addressing the quadratic self-attention cost. It introduces Test-Time Temporal Sampling (T3S), a training-free wrapper that generates multiple short, diverse subsequences, packs them into a single forward pass, and aggregates their logits, reducing computation to $O(L^2 \sum_{i=1}^m \alpha_i^2)$ with $\sum \alpha_i^2 < 1$. T3S combines multi-trial frame sampling and token subsampling with various logit-aggregation strategies, achieving robust accuracy gains (up to 3.1 percentage points on LongVideoBench) and about 2x speedups across multiple pretrained MLLMs without any training or architecture changes. Its plug-and-play nature and demonstrated compatibility across models and datasets offer a scalable path for efficient long-video understanding in real-world deployments.

Abstract

Processing long videos with multimodal large language models (MLLMs) poses a significant computational challenge, as the model's self-attention mechanism scales quadratically with the number of video tokens, resulting in high computational demand and slow inference speed. Current solutions, such as rule-based sub-sampling, learned frame selector, or memory-based summarization, often introduce their own trade-offs: they compromise accuracy, necessitate additional training, or decrease inference speed. In this paper, we propose Test-Time Temporal Sampling (T3S), a training-free, plug-and-play inference wrapper that enables MLLMs to process long videos both efficiently and effectively. T3S exploits spatiotemporal redundancy by generating multiple short and diverse subsequences of video tokens at inference time, packing them within a single forward pass, and aggregating their predictions. This multi-subsequence formulation broadens visual coverage while reducing the computational cost of self-attention from $O(L^2)$ to $O(\sum_{i=1}^m α_i^2L^2)$, where $\sum_{i=1}^m α_i^2 < 1$. Extensive experiments on long video understanding benchmarks demonstrate that T3S improves accuracy by up to 3.1% and reduces first token delay by $2.04\times$, all with minimal integration effort. Our approach operates entirely at inference time, requires no model modifications or fine-tuning, and is compatible with a wide range of pretrained MLLMs. T3S turns video redundancy into a computational advantage, offering a scalable solution for long-video understanding. The code is available at https://github.com/kaibinwang3/T3S.

Figures (4)

• Figure 1: Framework overview for Test-Time Temporal Sampling (T3S). The process begins by applying multi-trial frame sampling to a long input video, creating several shorter frame subsequences. These subsequences are then processed by a vision encoder to produce video embeddings. To manage sequence length, token subsampling is applied to these embeddings. The resulting shorter sequences are combined with textual inputs, and fed into the MLLM for parallel inference. Finally, the multiple outputs generated in parallel are combined through an aggregation step to produce the final prediction. This process is repeated autoregressively, feeding the aggregated output back into the model to generate subsequent tokens.
• Figure 2: Empirical speedup w.r.t.$\alpha_1$ and $\alpha_2$.
• Figure 3: Accuracy (%) as a function of the number of trials ($m$).
• Figure 4: Accuracy (%) as a function of the number of trials ($m$).