Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink

TL;DR

Mirage in the Eyes analyzes how attention sink in multimodal LLMs contributes to hallucinations and demonstrates a novel attack that manipulates attention sinks to induce erroneous content. The method does not rely on predefined targets and achieves high transferability, bypassing multiple mitigation strategies and affecting commercial APIs like GPT-4o and Gemini. By coupling attention dynamics with hidden-state embeddings, the attack increases hallucinated content while maintaining observable response quality, highlighting critical vulnerabilities in current instruction-tuning pipelines. The work motivates development of robust defenses such as adversarial purification and revised instruction-tuning to ensure faithful and reliable multimodal reasoning in real-world deployments.

Abstract

Fusing visual understanding into language generation, Multi-modal Large Language Models (MLLMs) are revolutionizing visual-language applications. Yet, these models are often plagued by the hallucination problem, which involves generating inaccurate objects, attributes, and relationships that do not match the visual content. In this work, we delve into the internal attention mechanisms of MLLMs to reveal the underlying causes of hallucination, exposing the inherent vulnerabilities in the instruction-tuning process. We propose a novel hallucination attack against MLLMs that exploits attention sink behaviors to trigger hallucinated content with minimal image-text relevance, posing a significant threat to critical downstream applications. Distinguished from previous adversarial methods that rely on fixed patterns, our approach generates dynamic, effective, and highly transferable visual adversarial inputs, without sacrificing the quality of model responses. Comprehensive experiments on 6 prominent MLLMs demonstrate the efficacy of our attack in compromising black-box MLLMs even with extensive mitigating mechanisms, as well as the promising results against cutting-edge commercial APIs, such as GPT-4o and Gemini 1.5. Our code is available at https://huggingface.co/RachelHGF/Mirage-in-the-Eyes.

Figures (17)

• Figure 1: An illustration of the attention sink phenomenon in MLLM responses. The sink token receives high attention scores in a columnar pattern. The hallucinated responses are marked bold with indigo tokens.
• Figure 2: Examples of inferred elements in ground truth responses: (a) overall summary of the image content, (b) overly detailed description of trivial objects, and (c) extra association not instructed by the task. The texts after "Human:" denote instructions, and those after "Assistant:" are ground truth responses. The examples are selected from the LLaVA-Instruct-150k liu2024visual and ALLaVA-4V chen2024allava datasets of detailed image caption and complex reasoning tasks, which are generated with GPT-4 achiam2023gpt and GPT-4V gpt4v models respectively.
• Figure 3: Per-sentence CLIPScore between input images and ground truth responses in instruction-tuning datasets. We report CLIPScore between input images and random response sentences as the baseline, denoted as random.
• Figure 4: (a)-(b) Per-sentence CLIPScore between input images and MLLM responses of InstructBLIP and LLaVA-1.5. (c) Mean CLIPScore of MLLM responses before and after the sink token. The postfix -beam, -greedy, and -nucleus represent beam search, greedy search, and nucleus sampling decoding respectively. The missing bars indicate no generated sentences of the corresponding length.
• Figure 5: Distribution of cosine similarity between multi-modal input embeddings and generated token embeddings. We compare the similarity of sink tokens (with the postfix -sink) and all other tokens (with the postfix -other).