Q Cache: Visual Attention is Valuable in Less than Half of Decode Layers for Multimodal Large Language Model

TL;DR

This work tackles the high inference cost of multimodal LLMs caused by abundant visual tokens and KV cache bottlenecks. It analyzes inter-layer attention redundancy and finds that adjacent decode layers exhibit highly similar patterns (with over half being effectively redundant), enabling cross-layer sharing through Lazy Attention and a dynamic Q Cache. The proposed Global Lazy Attention and Visual Lazy Attention modes render layer-wise KV cache reuse practical and compatible with Flash Attention, achieving about a 35% KV cache reduction and 1.5x–1.6x throughput with roughly 1% accuracy loss across several MLLMs, while remaining orthogonal to token-wise pruning methods. The results offer practical gains in efficiency and provide insights into attention redundancy that could guide future cache design and instruction-tuning strategies.

Abstract

Multimodal large language models (MLLMs) are plagued by exorbitant inference costs attributable to the profusion of visual tokens within the vision encoder. The redundant visual tokens engenders a substantial computational load and key-value (KV) cache footprint bottleneck. Existing approaches focus on token-wise optimization, leveraging diverse intricate token pruning techniques to eliminate non-crucial visual tokens. Nevertheless, these methods often unavoidably undermine the integrity of the KV cache, resulting in failures in long-text generation tasks. To this end, we conduct an in-depth investigation towards the attention mechanism of the model from a new perspective, and discern that attention within more than half of all decode layers are semantic similar. Upon this finding, we contend that the attention in certain layers can be streamlined by inheriting the attention from their preceding layers. Consequently, we propose Lazy Attention, an efficient attention mechanism that enables cross-layer sharing of similar attention patterns. It ingeniously reduces layer-wise redundant computation in attention. In Lazy Attention, we develop a novel layer-shared cache, Q Cache, tailored for MLLMs, which facilitates the reuse of queries across adjacent layers. In particular, Q Cache is lightweight and fully compatible with existing inference frameworks, including Flash Attention and KV cache. Additionally, our method is highly flexible as it is orthogonal to existing token-wise techniques and can be deployed independently or combined with token pruning approaches. Empirical evaluations on multiple benchmarks demonstrate that our method can reduce KV cache usage by over 35% and achieve 1.5x throughput improvement, while sacrificing only approximately 1% of performance on various MLLMs. Compared with SOTA token-wise methods, our technique achieves superior accuracy preservation.

Figures (8)

• Figure 1: A brief comparison on K Cache in varied decode layers of a MLLM. We introduce a new technique Q Cache, which significantly reduces the K Cache memory footprint. Compared to other methods, it is more flexible, plug-and-play, and compatible with Flash Attention.
• Figure 2: Jensen-Shannon (JS) divergence of the attention scores between each layer and its adjacent layer below it. The gray dashed boxes highlight the regions with values close to 0, where the attention score distributions in the two layers are nearly identical, suggesting that the adjacent decode layers share highly similar attention patterns.
• Figure 3: The diagram of the entire framework. Left: The optimized LLM component within a MLLM. It consists of several standard decode layers and multiple custom lazy blocks. Each lazy block contains one regular decode layer along with several lazy attention layers. Right: The internal structure of a lazy block. The distinctive characteristic of the lazy attention layer lies in its replacement of the naive attention with lazy attention (Global Lazy Attention or Visual Lazy Attention), which directly utilizes the queries and keys from the first decode layer within the block. To avoid the full computation typically associated with naive attention, the lazy attention focuses on eliminating redundant computations. This design helps to optimize memory cost while managing computational complexity effectively.
• Figure 4: A comprehensive visualization of the Q, K, V and caches in each layer of the lazy block in the prefilling and decoding stage under two modes. Tokens bounded by dashed lines indicate that they are not computed by the current layer or are not retained in the KV cache of this layer. The red dashed box highlights the main differences between GLA and VLA.
• Figure 5: The benefits of GLA and VLA over the Vanilla LLaVA-NEXT-7B model in parameters, KV cache, max batchsize and throughput in decoding phase.