Beyond Accuracy: Evaluating Grounded Visual Evidence in Thinking with Images

TL;DR

ViEBench tackles the problem of evaluating grounded visual reasoning in Thinking-with-Images VLMs beyond final accuracy by introducing a process-verifiable benchmark with 200 high-resolution images and expert-annotated visual evidence. It couples perception and reasoning tasks with a dual-axis IoA-based capability matrix to diagnose grounding and reasoning integrity, revealing that many models reach correct answers without faithful grounding and that some locate evidence yet fail to synthesize it into correct conclusions. The work contributes a rigorous annotation pipeline, a four-quadrant diagnostic framework, and extensive experimental results across agentic and non-agentic baselines, highlighting practical failure modes and guiding improvements for robust visual thinking. Overall, ViEBench provides a transparent, actionable benchmark to steer the development of truly grounded, reasoning-enabled agentic VLMs with potential broad impact on safety, reliability, and real-world deployment.

Abstract

Despite the remarkable progress of Vision-Language Models (VLMs) in adopting "Thinking-with-Images" capabilities, accurately evaluating the authenticity of their reasoning process remains a critical challenge. Existing benchmarks mainly rely on outcome-oriented accuracy, lacking the capability to assess whether models can accurately leverage fine-grained visual cues for multi-step reasoning. To address these limitations, we propose ViEBench, a process-verifiable benchmark designed to evaluate faithful visual reasoning. Comprising 200 multi-scenario high-resolution images with expert-annotated visual evidence, ViEBench uniquely categorizes tasks by difficulty into perception and reasoning dimensions, where reasoning tasks require utilizing localized visual details with prior knowledge. To establish comprehensive evaluation criteria, we introduce a dual-axis matrix that provides fine-grained metrics through four diagnostic quadrants, enabling transparent diagnosis of model behavior across varying task complexities. Our experiments yield several interesting observations: (1) VLMs can sometimes produce correct final answers despite grounding on irrelevant regions, and (2) they may successfully locate the correct evidence but still fail to utilize it to reach accurate conclusions. Our findings demonstrate that ViEBench can serve as a more explainable and practical benchmark for comprehensively evaluating the effectiveness agentic VLMs. The codes will be released at: https://github.com/Xuchen-Li/ViEBench.

Figures (7)

• Figure 1: Traditional benchmarks provide a superficial thinking evaluation by relying solely on final answer accuracy, which fails to detect if a model correctly answers via irrelevant visual regions. In contrast, ViEBench performs a faithful "Thinking-with-Images" audit by cross-referencing answer accuracy with visual grounding alignment. Our dual-axis capability matrix deconstructs VLMs performance into four fine-grained quadrants to provide a diagnostic map that identifies whether correct predictions are rooted in sound visual evidence.
• Figure 2: ViEBench audits the consistency between visual grounding and logical reasoning in agentic VLMs. By integrating expert-annotated BBox with a dual-axis evaluation protocol, we categorize model behaviors into four diagnostic metrics, providing a more rigorous assessment beyond accuracy only.
• Figure 3: Representative examples from ViEBench across four real-world scenarios. Each case illustrates a complex reasoning task where the critical evidence is spatially sparse and requires precise cropping to resolve.
• Figure 4: Scene distribution of the perception and reasoning categories in ViEBench.
• Figure 5: We visualize $IoA(B_{gt}, B_{pred})$ and $IoA(B_{pred}, B_{gt})$ across perception and reasoning tasks. The results reveal distinct strategies. (Inst. denotes Instruction-tuned)