PruneHal: Reducing Hallucinations in Multi-modal Large Language Models through Adaptive KV Cache Pruning
Fengyuan Sun, Hui Chen, Xinhao Xu, Dandan Zheng, Jingdong Chen, Jun Zhou, Jungong Han, Guiguang Ding
TL;DR
This work identifies a strong link between object hallucinations in multimodal LLMs and insufficient attention to visual tokens, attributed in part to visual token redundancy. It introduces PruneHal, a training-free adaptive KV cache pruning framework that preserves the most attended visual tokens to guide generation while maintaining language prior, and it is compatible with various decoding strategies for hallucination mitigation. Empirical results across multiple MLLMs and benchmarks show that PruneHal consistently reduces hallucinations (improving CHAIR and AMBER metrics) and enhances correctness under GPT-4V evaluation, with negligible or even reduced latency. The approach is model-agnostic, simple to integrate, and can be combined with existing mitigation techniques to achieve state-of-the-art performance with minimal computational overhead, enabling more reliable multimodal generation in practical settings.
Abstract
While multi-modal large language models (MLLMs) have made significant progress in recent years, the issue of hallucinations remains a major challenge. To mitigate this phenomenon, existing solutions either introduce additional data for further training or incorporate external or internal information during inference. However, these approaches inevitably introduce extra computational costs. In this paper, we observe that hallucinations in MLLMs are strongly associated with insufficient attention allocated to visual tokens. In particular, the presence of redundant visual tokens disperses the model's attention, preventing it from focusing on the most informative ones. As a result, critical visual cues are often under-attended, which in turn exacerbates the occurrence of hallucinations. Building on this observation, we propose \textbf{PruneHal}, a training-free, simple yet effective method that leverages adaptive KV cache pruning to enhance the model's focus on critical visual information, thereby mitigating hallucinations. To the best of our knowledge, we are the first to apply token pruning for hallucination mitigation in MLLMs. Notably, our method don't require additional training and incurs nearly no extra inference cost. Moreover, PruneHal is model-agnostic and can be seamlessly integrated with different decoding strategies, including those specifically designed for hallucination mitigation. We evaluate PruneHal on several widely used hallucination evaluation benchmarks using four mainstream MLLMs, achieving robust and outstanding results that highlight the effectiveness and superiority of our method. Our code will be publicly available.