Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models

TL;DR

MemVR introduces a memory-space visual retracing decoding paradigm for multimodal LLMs, reinjecting visual evidence into an intermediate FFN layer to mitigate hallucinations without extra training or iterations. By dynamically triggering visual retracing based on layer uncertainty and grounding predictions in refreshed visual tokens, MemVR improves factual alignment across multiple benchmarks while maintaining competitive latency. The approach is theoretically grounded in information bottleneck analysis and mutual-information arguments, and empirically shows strong gains on hallucination-specific and general-purpose tasks across diverse model families. The work offers a practical, plug-and-play solution for improving reliability of MLLMs in vision-rich tasks with wide applicability and minimal overhead.

Abstract

Despite their impressive capabilities, multimodal large language models (MLLMs) are prone to hallucinations, i.e., the generated content that is nonsensical or unfaithful to input sources. Unlike in LLMs, hallucinations in MLLMs often stem from the sensitivity of text decoder to visual tokens, leading to a phenomenon akin to "amnesia" about visual information. To address this issue, we propose MemVR, a novel decoding paradigm inspired by common cognition: when the memory of an image seen the moment before is forgotten, people will look at it again for factual answers. Following this principle, we treat visual tokens as supplementary evidence, re-injecting them into the MLLM through Feed Forward Network (FFN) as "key-value memory" at the middle trigger layer. This "look-twice" mechanism occurs when the model exhibits high uncertainty during inference, effectively enhancing factual alignment. Comprehensive experimental evaluations demonstrate that MemVR significantly mitigates hallucination across various MLLMs and excels in general benchmarks without incurring additional time overhead. The implementation is available from https://github.com/1zhou-Wang/MemVR

Figures (18)

• Figure 1: Comparison of the conventional CD-based hallucination mitigation paradigm VCD, and our proposed efficient MemVR.
• Figure 2: Radar charts comparing models across benchmarks.
• Figure 3: Uncertainty of different layers to predict the next token. Rows denote indices of the early layers, and column names are decoded tokens in each step. Uncertainty distribution is dynamic.
• Figure 4: (Left) Performance under different scaling ratios to text / image feature value on MME; (Right) Performance changes when look-twice to text / image / text+image (i.e, @t+i), respectively.
• Figure 5: Uncertainty distribution across layers during token reasoning in hallucinations. Red-outlined regions show higher uncertainty in middle and late layers for hallucinatory tokens: a, pom.

Theorems & Definitions (11)

• Theorem 4.1
• Theorem 4.2
• Theorem 4.3
• Theorem 1.1
• proof
• Theorem 1.2
• proof
• Theorem 1.3
• proof
• Theorem 1.4