ClueTracer: Question-to-Vision Clue Tracing for Training-Free Hallucination Suppression in Multimodal Reasoning

TL;DR

This work investigates hallucinations in Multimodal Large Reasoning Models (MLRMs) arising from reasoning drift during long-horizon inference. It introduces ClueRecall, a layer-wise visual-grounding metric, and ClueTracer, a training-free plug-in that traces the flow of task-relevant clues along the query→output→vision pathway to localize minimal, high-signal visual evidence and suppress spurious cues. By selecting an optimal grounding layer $L_{ m max}$ and identifying key query tokens via output-axis variability, ClueTracer constructs evidence-regions and supports a two-stage inference with focused visual crops, achieving consistent improvements on reasoning benchmarks (e.g., HallusionBench, VMCBench) and transferring to non-reasoning MLLMs (e.g., LLaVA, Qwen). The method is architecture-agnostic and training-free, offering a practical, deployment-friendly approach to reduce visual hallucinations while maintaining or improving reasoning performance, with average gains around the mid-teens in percentage points.

Abstract

Large multimodal reasoning models solve challenging visual problems via explicit long-chain inference: they gather visual clues from images and decode clues into textual tokens. Yet this capability also increases hallucinations, where the model generates content that is not supported by the input image or the question. To understand this failure mode, we identify \emph{reasoning drift}: during clue gathering, the model over-focuses on question-irrelevant entities, diluting focus on task-relevant cues and gradually decoupling the reasoning trace from visual grounding. As a consequence, many inference-time localization or intervention methods developed for non-reasoning models fail to pinpoint the true clues in reasoning settings. Motivated by these insights, we introduce ClueRecall, a metric for assessing visual clue retrieval, and present ClueTracer, a training-free, parameter-free, and architecture-agnostic plugin for hallucination suppression. ClueTracer starts from the question and traces how key clues propagate along the model's reasoning pathway (question $\rightarrow$ outputs $\rightarrow$ visual tokens), thereby localizing task-relevant patches while suppressing spurious attention to irrelevant regions. Remarkably, \textbf{without any additional training}, ClueTracer improves all \textbf{reasoning} architectures (including \texttt{R1-OneVision}, \texttt{Ocean-R1}, \texttt{MM-Eureka}, \emph{etc}.) by $\mathbf{1.21\times}$ on reasoning benchmarks. When transferred to \textbf{non-reasoning} settings, it yields a $\mathbf{1.14\times}$ gain.

Figures (7)

• Figure 1: (a) Example outputs from a reasoning model; hallucinated content is highlighted in red. (b) Reasoning drift in a multimodal reasoning model, where attention during inference is allocated to task-irrelevant regions. (c) An intuitive illustration: by progressively zooming in on task-critical regions, hallucinations (in red) diminish and the answers become correct. (d) Radar-chart comparisons among different reasoning models; ClueTracer is training-free and architecture-agnostic.
• Figure 2: Failure modes of prior mitigation: (a) bias accumulation and (b) clue neglect.
• Figure 3: (a) Question to Output Heat-map. (b) Visual to Output Heatmap. The dashed outline denotes expected attention region, consistent with the heatmap.
• Figure 4: Overview of our methods. Top: labeled data used by , the $\texttt{bbox}$ and $\texttt{cat}$ can either come from COCO annotations or a lightweight model, enabling our pipeline to extend to arbitrary datasets. Middle: computing layer-wise and selecting the layer $L_{\max}$ with the strongest visual clue retrieval. Bottom: follow the question $\rightarrow$ output $\rightarrow$ visual attention pathway to progressively localize task-relevant visual regions. Best viewed in color.
• Figure 5: (a) Comparative scatter plot of attention allocation versus output mentions for task-relevant objects; (b) case study showing that reduces reasoning drift; (c) case study showing that enhances fine-grained perceptual sensitivity.