Dynamic Token Reduction during Generation for Vision Language Models

TL;DR

This work tackles the quadratic complexity of decoder attention in Vision-Language Models by introducing Dynamic Rate (DyRate), a dynamic, attention-aware token pruning framework. DyRate uses a lightweight predictor to map attention distributions across token types to a current compression rate $R$, and enables differentiable, end-to-end training via Gumbel-Softmax sampling of $R$ among $K$ discrete values. By adaptively reducing visual tokens during autoregressive generation, DyRate achieves substantial FLOPs reductions while preserving generation quality across short and long responses, validated on multiple benchmarks with no manual rate tuning. The approach advances resource-aware multimodal generation, enabling efficient deployment in real-world scenarios with varying response lengths and content complexity.

Abstract

Vision-Language Models (VLMs) have achieved notable success in multimodal tasks but face practical limitations due to the quadratic complexity of decoder attention mechanisms and autoregressive generation. Existing methods like FASTV and VTW have achieved notable results in reducing redundant visual tokens, but these approaches focus on pruning tokens in a single forward pass without systematically analyzing the redundancy of visual tokens throughout the entire generation process. In this paper, we introduce a dynamic pruning strategy tailored for VLMs, namedDynamic Rate (DyRate), which progressively adjusts the compression rate during generation. Our analysis of the distribution of attention reveals that the importance of visual tokens decreases throughout the generation process, inspiring us to adopt a more aggressive compression rate. By integrating a lightweight predictor based on attention distribution, our approach enables flexible adjustment of pruning rates based on the attention distribution. Our experimental results demonstrate that our method not only reduces computational demands but also maintains the quality of responses.

Figures (4)

• Figure 1: As the VLMs continue to generate, the attention proportions among the four token types (system, image, instruction, response) fluctuate. Notably, as the number of iterations increases, visual tokens receive decreasing attention. This observation suggests that we can adjust the appropriate compression rate $R$ based on the attention distribution to implement more aggressive pruning during the model's generation process.
• Figure 2: Attention stack per iteration during the decoding process of LLaVAv1.5-7B on the Flickr30K flickr30k dataset, the x-axis represents the time step in generations. The left graph depicts shallow layers, while the right graph represents deep layers. Our findings indicate that as generations progress, the importance of visual tokens gradually decreases. We categorize the input tokens at each iteration into four types: system prompt (sys), image token (img), user instruction (ins), and response token (res).
• Figure 3: The pipeline of our method. We calculate the attention distribution for each attentional head and train a linear classifier to find the optimal token pruning rate $R$. The classifier can be trained end-to-end.
• Figure 4: We explore the mask of the image for each token iteration in the model generation process.