Dynamic Token Compression for Efficient Video Understanding through Reinforcement Learning

Abstract

Multimodal Large Language Models have demonstrated remarkable capabilities in video understanding, yet face prohibitive computational costs and performance degradation from ''context rot'' due to massive visual token redundancy. Existing compression strategies typically rely on heuristics or fixed transformations that are often decoupled from the downstream task objectives, limiting their adaptability and effectiveness. To address this, we propose SCORE (Surprise-augmented token COmpression via REinforcement learning), a unified framework that learns an adaptive token compression policy. SCORE introduces a lightweight policy network conditioned on a surprise-augmented state representation that incorporates inter-frame residuals to explicitly capture temporal dynamics and motion saliency. We optimize this policy using a group-wise reinforcement learning scheme with a split-advantage estimator, stabilized by a two-stage curriculum transferring from static pseudo-videos to real dynamic videos. Extensive experiments on diverse video understanding benchmarks demonstrate that SCORE significantly outperforms state-of-the-art baselines. Notably, SCORE achieves a 16x prefill speedup while preserving 99.5% of original performance at a 10% retention ratio, offering a scalable solution for efficient long-form video understanding.

Figures (5)

• Figure 1: (a) Visual token redundancy induces context rot in video MLLMs. (b) A comparision of existing token compression methods and SCORE.
• Figure 2: SCORE pipeline. A lightweight visual compressor is inserted between the frozen vision encoder and the frozen LLM. During training, we optimize a surprise-augmented Bernoulli gating policy with group rollouts and an accuracy--sparsity reward. During inference, we use deterministic global top-$K$ selection to meet a target budget.
• Figure 3: Asymmetric advantage for group rollouts. We separate rollouts into a safe zone ($\Delta \mathrm{CE}>0$) and a penalty zone ($\Delta \mathrm{CE}\le 0$). Successful rollouts are ranked by sparsity via normalized advantages, while violating rollouts receive sparsity-weighted penalties to recover performance.
• Figure 4: Pseudo-video samples for curriculum warm-up. We construct short clips by repeating each sampled image for several frames and concatenate their captions into a single training target. This synthesis produces near-zero inter-frame residuals within repeated segments and sharp changes at image boundaries, offering high-contrast temporal signals for learning redundancy-aware token pruning.
• Figure 5: Qualitative visualization of SCORE token masks. Top two rows: six pseudo-video inputs constructed from static images, where SCORE concentrates kept tokens on salient objects and action-related regions while pruning static backgrounds. Bottom two rows: two real-video examples, showing temporally adaptive masks that shift as motion and scene content evolve.