DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning

TL;DR

DeFacto tackles the problem of faithful reasoning in multimodal vision-language systems by enforcing consistency between the evidence used to justify an answer and the answer itself. It introduces a counterfactual thinking framework with positive, counterfactual abstention, and random-masking training paradigms, trained with Group Relative Policy Optimization (GRPO) using a composite reward that includes answer correctness, format conformity, and region-level coherence. The evidence dataset DeFacto-100K and a small faithfulness benchmark DeFacto-1.5K are built to enable automatic evaluation of region-level grounding. Experiments across general, document, and text-centric benchmarks show improvements in both answering accuracy and grounding faithfulness, demonstrating robust, interpretable multimodal reasoning; code and dataset are released.

Abstract

Recent advances in multimodal language models (MLLMs) have achieved remarkable progress in vision-language reasoning, especially with the emergence of "thinking with images," which integrates explicit visual steps into the reasoning process. While this paradigm strengthens image-based reasoning, a significant challenge remains: models may arrive at correct answers by relying on irrelevant or spurious regions, driven by prior knowledge or dataset biases. Even when the answer is correct, flawed reasoning indicates that the model has not truly understood the image, highlighting the critical importance of reasoning fidelity in multimodal tasks. To address this issue, we propose DeFacto, a counterfactual reasoning framework that jointly enforces accurate answering and faithful reasoning. A key component of our approach is the design of three complementary training paradigms: (i) positive, (ii) counterfactual, and (iii) random-masking. To enable these paradigms, we develop a pipeline that automatically localizes question-relevant evidence and constructs positive, counterfactual, and random variants, resulting in a dataset of about 100k images. Building on this framework, we train multimodal language models with GRPO-based reinforcement learning, where we design three complementary rewards to guide the model toward accurate answering and evidence-grounded reasoning. Experiments on diverse benchmarks demonstrate that DeFacto substantially improves both answer accuracy and reasoning faithfulness, establishing a stronger foundation for interpretable multimodal reasoning. The code is available on GitHub and the dataset is released on HuggingFace.

Figures (3)

• Figure 1: Qualitative examples of unfaithful reasoning in thinking-with-images. We categorize failures into three types: Mislocalized Failure (wrong evidence, wrong answer), Spurious Correctness (wrong evidence, correct answer), and Faithful Incorrectness (correct evidence, wrong answer). We compare two representative baselines (Deepeyes and GRIT) with our method on three cases.
• Figure 2: An overview of our counterfactual framework with three inputs: positive (full evidence), counterfactual (masked evidence), and random (masked irrelevant regions).
• Figure 3: Heatmap visualization of faithful reasoning metrics across DeFacto-1.5K.