Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models

TL;DR

This work tackles the challenge of long-video understanding in multimodal language models by introducing QTSplus, a query-aware token selector that dynamically gates visual tokens based on a textual query and video statistics. The method scores tokens via cross-attention, predicts an instance-specific retention budget $\rho$, and uses a differentiable gate during training with a lightweight re-encoder to preserve temporal order, significantly cutting the KV-cache and attention burden. Empirically, integrating QTSplus with Qwen2.5-VL yields up to $89\%$ reduction in visual tokens and up to $28\%$ end-to-end latency on long videos, while achieving near-parity or improvements on multiple long-video benchmarks, especially for temporally focused tasks. The results demonstrate that adaptive, relevance-aware tokenization is a practical path to scaling MLLMs to hour-long inputs under realistic compute and memory constraints, with potential extensions to streaming scenarios and multi-query/multi-camera settings.

Abstract

Despite the recent advances in the video understanding ability of multimodal large language models (MLLMs), long video understanding remains a challenge. One of the main issues is that the number of vision tokens grows linearly with video length, which causes an explosion in attention cost, memory, and latency. To solve this challenge, we present Query-aware Token Selector (\textbf{QTSplus}), a lightweight yet powerful visual token selection module that serves as an information gate between the vision encoder and LLMs. Given a text query and video tokens, QTSplus dynamically selects the most important visual evidence for the input text query by (i) scoring visual tokens via cross-attention, (ii) \emph{predicting} an instance-specific retention budget based on the complexity of the query, and (iii) \emph{selecting} Top-$n$ tokens with a differentiable straight-through estimator during training and a hard gate at inference. Furthermore, a small re-encoder preserves temporal order using absolute time information, enabling second-level localization while maintaining global coverage. Integrated into Qwen2.5-VL, QTSplus compresses the vision stream by up to \textbf{89\%} and reduces end-to-end latency by \textbf{28\%} on long videos. The evaluation on eight long video understanding benchmarks shows near-parity accuracy overall when compared with the original Qwen models and outperforms the original model by \textbf{+20.5} and \textbf{+5.6} points respectively on TempCompass direction and order accuracies. These results show that QTSplus is an effective, general mechanism for scaling MLLMs to real-world long-video scenarios while preserving task-relevant evidence.

Figures (6)

• Figure 1: The illustration of our proposed QTSplus. The improvement is applied to the step of multimodal token mixture.
• Figure 2: Comparison of the Training Processes for QTSplus-3B and QTSplus-3B-FT.
• Figure 3: Data processing pipeline. VSCQ denotes visual single-choice questions. Starting from base video captions, Qwen3‑235B (text-only) synthesizes multiple-choice items (QTS-VSCQ1). Qwen2.5‑VL then (1) screens QTS‑VSCQ1 by answering each item and keeping only those it judges correctly (QTS‑VSCQ2); and (2) generates free-form answers given the video and question to form a visual QA corpus (QTS‑VQA). The two teacher‑verified sets supervise the student through distillation.
• Figure 4: Scaling characteristics on a single A100 GPU (FlashAttentiondao2022flashattentionfastmemoryefficientexact enabled). Left: number of vision embeddings forwarded to the LM versus frame count. Right: end‑to‑end latency to first complete output versus frame count. QTSplus‑3B‑FT consistently reduces the embedding budget and latency relative to Qwen2.5‑VL‑3B‑Instruct.
• Figure 5: Correlation between budget-head inputs and predicted budget ratio $\rho$. Each subplot shows the relationship between one scalar input to the budget head and the predicted retention ratio $\rho$: (top left) mean query embedding $s_q$, (top right) $\log M$ (logarithm of the number of visual tokens), (bottom left) $r_{\max}$ (maximum query--vision relevance), and (bottom right) $H(p)$ (entropy of the normalised relevance distribution). A red dashed line indicates a linear fit; the reported value in the title is the corresponding correlation coefficient.