Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning

Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, Khyathi Raghavi Chandu

TL;DR

This work addresses over-abstention in vision-language selective prediction by introducing ReCoVERR, an inference-time algorithm that collects high-confidence visual evidences through LLM-driven questioning to verify uncertain predictions. ReCoVERR iteratively generates sub-questions, retains reliable and relevant evidences, and uses a Natural Language Inference step to determine sufficiency before answering, all without extra training. Across A-OKVQA and VQAv2, and for multiple VLMs (BLIP2, InstructBLIP, LLaVA-1.5), ReCoVERR increases coverage by up to 20% while maintaining the specified risk threshold, with notable gains for less-task-fine-tuned models. Ablation and transfer experiments show that evidence reliability and relevance are crucial, and that the approach can generalize to new tasks with limited tuning.

Abstract

Selective prediction minimizes incorrect predictions from vision-language models (VLMs) by allowing them to abstain from answering when uncertain. However, when deploying a vision-language system with low tolerance for inaccurate predictions, selective prediction may be over-cautious and abstain too frequently, even on many correct predictions. We introduce ReCoVERR, an inference-time algorithm to reduce the over-abstention of a selective vision-language system without increasing the error rate of the system's predictions. When the VLM makes a low-confidence prediction, instead of abstaining ReCoVERR tries to find relevant clues in the image that provide additional evidence for the prediction. ReCoVERR uses an LLM to pose related questions to the VLM, collects high-confidence evidences, and if enough evidence confirms the prediction the system makes a prediction instead of abstaining. ReCoVERR enables three VLMs (BLIP2, InstructBLIP, and LLaVA-1.5) to answer up to 20% more questions on the VQAv2 and A-OKVQA tasks without decreasing system accuracy, thus improving overall system reliability. Our code is available at https://github.com/tejas1995/ReCoVERR.

Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning

TL;DR

This work addresses over-abstention in vision-language selective prediction by introducing ReCoVERR, an inference-time algorithm that collects high-confidence visual evidences through LLM-driven questioning to verify uncertain predictions. ReCoVERR iteratively generates sub-questions, retains reliable and relevant evidences, and uses a Natural Language Inference step to determine sufficiency before answering, all without extra training. Across A-OKVQA and VQAv2, and for multiple VLMs (BLIP2, InstructBLIP, LLaVA-1.5), ReCoVERR increases coverage by up to 20% while maintaining the specified risk threshold, with notable gains for less-task-fine-tuned models. Ablation and transfer experiments show that evidence reliability and relevance are crucial, and that the approach can generalize to new tasks with limited tuning.

Abstract

Selective prediction minimizes incorrect predictions from vision-language models (VLMs) by allowing them to abstain from answering when uncertain. However, when deploying a vision-language system with low tolerance for inaccurate predictions, selective prediction may be over-cautious and abstain too frequently, even on many correct predictions. We introduce ReCoVERR, an inference-time algorithm to reduce the over-abstention of a selective vision-language system without increasing the error rate of the system's predictions. When the VLM makes a low-confidence prediction, instead of abstaining ReCoVERR tries to find relevant clues in the image that provide additional evidence for the prediction. ReCoVERR uses an LLM to pose related questions to the VLM, collects high-confidence evidences, and if enough evidence confirms the prediction the system makes a prediction instead of abstaining. ReCoVERR enables three VLMs (BLIP2, InstructBLIP, and LLaVA-1.5) to answer up to 20% more questions on the VQAv2 and A-OKVQA tasks without decreasing system accuracy, thus improving overall system reliability. Our code is available at https://github.com/tejas1995/ReCoVERR.
Paper Structure (29 sections, 7 equations, 4 figures, 8 tables, 1 algorithm)

This paper contains 29 sections, 7 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Multimodal Selective Prediction
  3. Evaluating Selective Predictors
  4. ReCoVERR
  5. Initialize Evidences from Vision Tools:
  6. Generating Visual Evidences:
  7. Check Evidence Reliability:
  8. Check Evidence Relevance:
  9. Check Sufficiency of Collected Evidences:
  10. Experiments
  11. Vision-Language Reasoning Tasks: A-OKVQA and VQAv2
  12. Vision-Language Models
  13. Calibrated VLM Confidence Estimates
  14. ReCoVERR Implementation Details
  15. ReCoVERR Model-based Tools
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Illustration of ReCoVERR. The VLM predicts that the floor has two tile colors with low confidence. Instead of abstaining, ReCoVERR collects reliable and relevant visual evidences related to the question. ReCoVERR makes salient the evidence that the floor tiles are red and white, helping to verify the VLM's original answer.
  • Figure 2: The ReCoVERR algorithm. If the VLM is uncertain in its prediction (1), ReCoVERR tries to verify the VLM hypothesis by collecting evidences. ReCoVERR undertakes multiple turns of evidence collection, which involves generating visual evidences by using an LLM to ask questions to the VLM (2), retaining the reliable (3) and relevant (4) evidences, and checking whether the collected evidence entails the hypothesis (5).
  • Figure 3: Calibration curves for BLIP-2, InstructBLIP and LLaVA-1.5 on A-OKVQA questions.
  • Figure 4: Distribution of VLM prediction confidences on A-OKVQA calibration set.