Variational Visual Question Answering for Uncertainty-Aware Selective Prediction

TL;DR

This work tackles the unreliability of vision-language models in VQA by introducing Variational VQA (VarVQA), which learns a posterior over model parameters via the IVON optimizer to enable intrinsic uncertainty estimation. By using MC sampling for predictions and a novel risk-averse selector, VarVQA achieves superior calibration and high-stakes selective prediction compared to AdamW and MC Dropout, while maintaining competitive accuracy. The approach demonstrates strong robustness to distribution shift (ID/OOD) and improves reliability with fewer inference samples, establishing variational learning as a practical route to safer, more trustworthy multimodal models. The framework is validated across large VLMs (ViLT, BEiT-3) and tasks (VQAv2, NLVR2, AdVQA), with extensive analysis of calibration, sampling, and selector strategies, and identifiable avenues for further efficiency and integration with other uncertainty-estimation methods.

Abstract

Despite remarkable progress in recent years, vision language models (VLMs) remain prone to overconfidence and hallucinations on tasks such as Visual Question Answering (VQA) and Visual Reasoning. Bayesian methods can potentially improve reliability by helping models selectively predict, that is, models respond only when they are sufficiently confident. Unfortunately, Bayesian methods are often assumed to be costly and ineffective for large models, and so far there exists little evidence to show otherwise, especially for multimodal applications. Here, we show the effectiveness and competitive edge of variational Bayes for selective prediction in VQA for the first time. We build on recent advances in variational methods for deep learning and propose an extension called "Variational VQA". This method improves calibration and yields significant gains for selective prediction on VQA and Visual Reasoning, particularly when the error tolerance is low ($\leq 1\%$). Often, just one posterior sample can yield more reliable answers than those obtained by models trained with AdamW. In addition, we propose a new risk-averse selector that outperforms standard sample averaging by considering the variance of predictions. Overall, we present compelling evidence that variational learning is a viable option to make large VLMs safer and more trustworthy.

Figures (20)

• Figure 1: Despite recent performance gains, VLMs trained with popular optimizers like AdamW do not know when they are wrong. Our Variational VQA approach uses learned parameter variances to enable models to abstain when uncertain. The example is from BEiT-3 beit3, which achieves near-human accuracy on VQAv2 vqav2.
• Figure 2: Overview of the methods we experiment with and their selectors. Variational VQA employs $g^{\mu}_{\textrm{MP}}$ for calibration and $g_{\textrm{BPA}}$ for selective prediction.
• Figure 3: Results on Accuracy, calibration and selective prediction on VQAv2 after fine-tuning.
• Figure 4: Comparison of Variational VQA to MC Dropout, an approximate variational method that uses the same inference compute, on the high-stakes selective prediction metrics.
• Figure 5: Accuracy, calibration and selective prediction results for different VQAv2/AdVQA mixtures for BEiT-3 large. In (d), every model in the gray area is worse than a model that abstains on every input.