Selectively Answering Visual Questions

TL;DR

This study addresses the problem of calibrated, selective answering in visual question answering (VQA) by evaluating calibration methods for large multimodal models (LMMs) and large language models (LLMs) on VizWiz-VQA and UNK-VQA. It analyzes four scoring strategies—Likelihood, Sampling Repetition, Sampling Diversity, and the proposed Avg BLEU—to estimate confidence and abstention thresholds in in-context learning settings. The key finding is that visually grounded LMMs often calibrate better than text-only LLMs on several metrics, and Avg BLEU provides a robust, unified measure that combines the strengths of sampling and likelihood approaches across modalities. The results have practical significance for assistive VQA applications, enabling more reliable uncertainty estimates, while acknowledging limitations related to dataset scope and caption quality.

Abstract

Recently, large multi-modal models (LMMs) have emerged with the capacity to perform vision tasks such as captioning and visual question answering (VQA) with unprecedented accuracy. Applications such as helping the blind or visually impaired have a critical need for precise answers. It is specially important for models to be well calibrated and be able to quantify their uncertainty in order to selectively decide when to answer and when to abstain or ask for clarifications. We perform the first in-depth analysis of calibration methods and metrics for VQA with in-context learning LMMs. Studying VQA on two answerability benchmarks, we show that the likelihood score of visually grounded models is better calibrated than in their text-only counterparts for in-context learning, where sampling based methods are generally superior, but no clear winner arises. We propose Avg BLEU, a calibration score combining the benefits of both sampling and likelihood methods across modalities.

Figures (7)

• Figure 1: Sampled outputs from an LMM on a VizWiz-VQA example gurari2018vizwiz are used to measure model calibration. In this paper we contrast the LMM calibration results against LLMs that only sees the image caption. Sampling based methods struggle to measure uncertainty, motivating our proposed Avg BLEU as a confidence estimate.
• Figure 2: Unanswerable questions in VizWiz-VQA. Many are not answerable due to low quality images. Sometimes the intent of the question poser, and therefore the correct answer, cannot be inferred.
• Figure 3: LLM 4-shot prompting for VizWiz-VQA using captions from vizwizcaps2020.
• Figure 4: LMM 4-shot prompting for VizWiz-VQA.
• Figure 5: Question "Alright see if you can see the ORCA serial number now.". The top 3 answers by PaLM2 Bison with their likelihood are ('31631036', -1.32), ('468563995', -1.41), ('4667926374', -1.41). They are neither in the image nor in the gold captions. The serial number is not in the image. Correctly, LLaVA 13B does not trigger a response in this case considering the question unanswerable.