VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoning

TL;DR

VISCO introduces the first benchmark for fine-grained critique and correction in visual reasoning LVLMs, enabling self-improvement through dense per-step feedback and natural language explanations. It collects 1645 QA pairs with 5604 step annotations across 18 datasets and 8 tasks, and evaluates 24 LVLMs using hierarchical F1-based critique plus a geometric VISCore metric, alongside a correction task quantified by PCR and NCR. The study finds human-written critiques substantially boost correction (up to 76%), while model-generated critiques are a major bottleneck, and identifies three critique failure patterns. The LookBack strategy, which re-answers CoT steps by explicitly verifying information against the image, significantly improves critique (up to 13.5%) and correction (up to 11.5%), especially in perception tasks, offering a practical path toward reliable self-improvement in visual reasoning systems.

Abstract

The ability of large vision-language models (LVLMs) to critique and correct their reasoning is an essential building block towards their self-improvement. However, a systematic analysis of such capabilities in LVLMs is still lacking. We propose VISCO, the first benchmark to extensively analyze the fine-grained critique and correction capabilities of LVLMs. Compared to existing work that uses a single scalar value to critique the entire reasoning [4], VISCO features dense and fine-grained critique, requiring LVLMs to evaluate the correctness of each step in the chain-of-thought and provide natural language explanations to support their judgments. Extensive evaluation of 24 LVLMs demonstrates that human-written critiques significantly enhance the performance after correction, showcasing the potential of the self-improvement strategy. However, the model-generated critiques are less helpful and sometimes detrimental to the performance, suggesting that critique is the crucial bottleneck. We identified three common patterns in critique failures: failure to critique visual perception, reluctance to "say no", and exaggerated assumption of error propagation. To address these issues, we propose an effective LookBack strategy that revisits the image to verify each piece of information in the initial reasoning. LookBack significantly improves critique and correction performance by up to 13.5%.

Figures (29)

• Figure 1: Evaluation settings of VISCO. We evaluate the self-improvement setting where LVLMs correct their own answers based on self-generated critique, and a human-assisted setting where correction is based on human-written critique.
• Figure 2: Overview of VISCO task, data and evaluation. VISCO dataset features step-wise critique with natural language explanations.
• Figure 3: Categories, tasks and datasets distribution of VISCO.
• Figure 4: Scaling curve for critique task.
• Figure 5: Scaling curve for answer-level F1.