Efficient Inference of Vision Instruction-Following Models with Elastic Cache

TL;DR

This work tackles the memory and latency challenges of KV caches in multimodal instruction-following LVLMs by introducing Elastic Cache, a training-free, plug-and-play approach that separates acceleration strategies for instruction encoding and output generation. It relies on an importance-driven, layer-wise cache merging to prune redundancy during encoding and a fixed-point elimination strategy to bound cache growth during generation, guided by key equations such as $N_I = \gamma \cdot T$ and moment-to-moment retention controlled by a truncation point $N_{tl}$. Empirical results on LVLMs (e.g., LLaVA-1.5, Qwen-VL) show substantial speedups (up to approximately $77.9\%$ actual speedup) while maintaining or improving PPL and ROUGE metrics, outperforming baselines like H2O and Local across various KV budgets. The method demonstrates robustness for long, multimodal instruction-following tasks and offers practical impact for real-world systems by reducing memory demands without retraining, though it relies on attention-based importance signals and raises considerations for societal impact in deployment.

Abstract

In the field of instruction-following large vision-language models (LVLMs), the efficient deployment of these models faces challenges, notably due to the high memory demands of their key-value (KV) caches. Conventional cache management strategies for LLMs focus on cache eviction, which often fails to address the specific needs of multimodal instruction-following models. Recognizing this gap, in this paper, we introduce Elastic Cache, a novel approach that benefits from applying distinct acceleration methods for instruction encoding and output generation stages. We investigate the metrics of importance in different stages and propose an importance-driven cache merging strategy to prune redundancy caches. Instead of discarding less important caches, our strategy identifies important key/value vectors as anchor points. Surrounding less important caches are then merged with these anchors, enhancing the preservation of contextual information in the KV caches while yielding an arbitrary acceleration ratio. For instruction encoding, we utilize the frequency to evaluate the importance of caches. Regarding output generation, we prioritize tokens based on their distance with an offset, by which both the initial and most recent tokens are retained. Results on a range of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also notably outperforms existing pruning methods in language generation across various tasks. Code is available at https://github.com/liuzuyan/ElasticCache

Figures (4)

• Figure 1: The main idea of Elastic Cache. Instruction encoding accounts for most of the theoretical computation cost, while the actual latency is negligible (here we use generating 512 tokens based on a 1024-token instruction as an example). This underscores that it's not just model weights but also the KV cache used in output generation that can become a significant bottleneck. We propose Elastic Cache through a Cache Merging based on the importance scores of instruction tokens, complemented by a fixed-point elimination strategy in the output generation phase. Our designs yield significant inference acceleration while maintaining generation quality.
• Figure 2: Results on visual instruct-following tasks. We evaluate Elastic Cache together with baselines on PPL (lower better) and ROUGE (higher better) metrics. We conduct LLaVA-1.5 of different sizes (a),(b) and Qwen-VL-7B(c) for visual tasks. Our Elastic Cache outperforms baselines consistently.
• Figure 3: Effects of the fixed-point elimination. We observe that fixing the elimination at the middle of the KV-Cache of the instruction attention score leads to better performance.
• Figure 4: Chat generation demo. We set the KV-Cache Budget as 0.5 and generate a response under the dialogue setting. The LLM generates meaningful and abundant outputs.