ViFP: A Framework for Visual False Positive Detection to Enhance Reasoning Reliability in VLMs

TL;DR

This work tackles false-positive reasoning in vision-language models by introducing ViFP, a framework that constructs targeted multi-step reasoning chains and detects FP reasoning through cross-checks between direct and multi-step outputs. It then corrects FP paths via information-gain–driven fine-tuning of reasoning templates and question types, yielding more reliable reasoning and higher accuracy across three real-world VQA datasets. A novel reliability metric, VoC, integrates improvement in multi-step accuracy with FP reduction to quantify the value of FP correction. The approach demonstrates consistent gains on closed-source VLMs and provides a modular, training-free pathway to enhance reasoning reliability in VLM-based VQA tasks, with practical implications for deploying trustworthy visual reasoning systems.

Abstract

During reasoning in vision-language models (VLMs), false positive (FP) reasoning occurs when a model produces the correct answer but follows an incorrect reasoning path, resulting in undermined reasoning reliability. Existing approaches mainly rely on prompt engineering, knowledge distillation or reinforcement learning to improve reasoning reliability, both of which require large amounts of high-quality data and thus limit practical applicability. Few approaches have focused on directly detecting and correcting FPs. To address these issues, we propose ViFP, a framework for Visual False Positive Detection to Enhance Reasoning Reliability in VLMs. ViFP builds effective reasoning paths through multi-turn QA and dynamically analyzes the consistency of the reasoning path to identify potential FPs. It also introduces a targeted reasoning chain correction mechanism to modify FP reasoning, thereby improving logical consistency and accuracy. Finally, we introduce a reliability evaluation metric, VoC, which integrates answer accuracy and the FP rate, providing a quantitative tool to assess whether a VLM not only answers correctly but also reasons reliably. Our experiments on closed-source VLMs show that ViFP consistently improves performance across three datasets: A-OKVQA, OK-VQA, and FVQA. On A-OKVQA, ViFP improves accuracy by up to 5.4%, surpassing the previous state-of-the-art by 4.3%, and significantly reduces the number of FPs, validating its benefits in enhancing reasoning reliability.

Figures (13)

• Figure 1: FP instance in visual reasoning. The VLM follows the same reasoning path twice (sun$\rightarrow$building$\rightarrow$shadows) but produces different answers (afternoon and noon).
• Figure 2: The overview framework of ViFP. (1) Question Classification: real-world VQA questions are classified along the dimensions of perception and reasoning, forming the initial question types. (2) Reasoning Chain Construction: the VLM summarizes reasoning chain templates for use in multi-step reasoning. (3) FP Detection: detecting FPs by comparing the consistency between direct and multi-step reasoning. (4) $QT$ and $Sq_i$ Fine-Tuning: Based on the detected FPs, the VLM refines the question types and the corresponding reasoning chains. (5) ViFP: Given an input (image question pair), ViFP conducts multi-step reasoning and iteratively refines its next reasoning to produce more accurate answers.
• Figure 3: Data distribution of question types on datasets
• Figure 4: Difficulty of different question types on datasets
• Figure 5: Improvement of ViFP-base VLMs