Visual Grounding Methods for VQA are Working for the Wrong Reasons!

TL;DR

This paper interrogates visual-grounding-based bias mitigation in VQA, arguing that reported gains arise from regularization that forgets linguistic priors rather than from true grounding. It demonstrates that training with irrelevant or even random visual cues yields similar improvements to cue-based methods, and shows that differences between grounding variants are not statistically significant. A simple, annotation-free regularizer that zeros out the ground-truth answers achieves near state-of-the-art on VQA-CPv2, reinforcing the claim that prior gains are regularization-driven. The authors advocate for rigorous grounding evaluation, reporting both train and test performance, and propose synthetic or grounded datasets to better assess true visual grounding effects. Overall, the work challenges the assumed link between grounding cues and genuine grounding, and calls for more robust benchmarks and evaluation frameworks for grounding in VQA.

Abstract

Existing Visual Question Answering (VQA) methods tend to exploit dataset biases and spurious statistical correlations, instead of producing right answers for the right reasons. To address this issue, recent bias mitigation methods for VQA propose to incorporate visual cues (e.g., human attention maps) to better ground the VQA models, showcasing impressive gains. However, we show that the performance improvements are not a result of improved visual grounding, but a regularization effect which prevents over-fitting to linguistic priors. For instance, we find that it is not actually necessary to provide proper, human-based cues; random, insensible cues also result in similar improvements. Based on this observation, we propose a simpler regularization scheme that does not require any external annotations and yet achieves near state-of-the-art performance on VQA-CPv2.

Figures (6)

• Figure 1: We find that existing visual sensitivity enhancement methods improve performance on VQA-CPv2 through regularization as opposed to proper visual grounding.
• Figure 2: Accuracies for HINT and SCR on VQAv2's val set, when fine-tuned either on the full train set or on the subset containing visual cues.
• Figure A3: Visualizations of most sensitive visual regions used by different variants of HINT to make predictions. We pick samples where all variants produce correct response to the question. The first column shows ground truth regions and columns 2-4 show visualizations from HINT trained on relevant, irrelevant and fixed random regions respectively.
• Figure A4: % CPIG for baseline and different variants of HINT and our method, computed using ground truth relevant regions taken from human attention maps (lower is better).
• Figure A5: % CPIG for baseline and different variants of SCR and our method, computed using ground truth relevant regions taken from textual explanations (txt).