AdaTP: Attention-Debiased Token Pruning for Video Large Language Models

TL;DR

Video LLMs incur high computation from abundant visual tokens, exacerbated by attention biases. AdaTP introduces a training-free token pruning pipeline with Global Debiasing and Local Debiasing modules and segment-aware progressive pruning to address global and local biases, respectively. By partitioning videos into segments via cosine similarity and leveraging text relevance plus spatial deduplication, AdaTP preserves semantically important content with significantly reduced FLOPs, achieving state-of-the-art results on multiple benchmarks and often matching or surpassing vanilla performance at reduced cost. The approach demonstrates robust efficiency gains across diverse Video LLMs (e.g., LLaVA-OneVision, LLaVA-Video) and datasets, offering a practical path to scalable video understanding without additional training.

Abstract

Video Large Language Models (Video LLMs) have achieved remarkable results in video understanding tasks. However, they often suffer from heavy computational overhead due to the large number of visual tokens generated from multiple video frames. Existing visual token compression methods often rely on attention scores from language models as guidance. However, these scores exhibit inherent biases: global bias reflects a tendency to focus on the two ends of the visual token sequence, while local bias leads to an over-concentration on the same spatial positions across different frames. To address the issue of attention bias, we propose $\textbf{A}$ttention-$\textbf{D}$ebi$\textbf{a}$sed $\textbf{T}$oken $\textbf{P}$runing for Video Large Language Models ($\textbf{AdaTP}$), a novel token pruning pipeline for Video LLMs. AdaTP integrates two dedicated debiasing modules into the pipeline, targeting global attention bias and local attention bias, respectively. Without the need for additional training, our method significantly reduces the computational overhead of Video LLMs while retaining the performance of vanilla models. Extensive evaluation shows that AdaTP achieves state-of-the-art performance in various commonly used video understanding benchmarks. In particular, on LLaVA-OneVision-7B, AdaTP maintains performance without degradation while using only up to $27.3\%$ FLOPs compared to the vanilla model. Our code will be released soon.

Figures (7)

• Figure 1: Visualization of attention bias. The visual tokens with top-10% attention scores are highlighted in red and green, where red tokens represent global attention bias, and green tokens represent local attention bias.
• Figure 2: Visualization of global attention bias in Layer 1 and 10. Each bar represents the sum of attention scores of visual tokens within a single frame.
• Figure 3: Visualization of local attention bias in Layer 1 and 10. Each block illustrates the sum of attention scores from prompt tokens to each spatial position across all frames.
• Figure 4: Illustration of our AdaTP pipeline. AdaTP mitigates attention bias by selectively retaining critical tokens based on text relevance and spatial diversity. The Global Debiasing Module identifies significant video segments, and the Local Debiasing Module further alleviates intra-segment spatial redundancy.
• Figure 5: Visualization example sampled from the VideoMME dataset. We adopt attention scores from layer 1 to generate the visualization results, in which we perform significant token pruning.