Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization

TL;DR

This work interrogates whetherVision-and-Language models truly learn general VQA skills or merely exploit dataset-specific cues by evaluating them across cross-dataset, out-of-distribution settings. It systematically compares discriminative and generative variants of ViLBERT and ALBEF on five VQA benchmarks, exploring the impact of multimodal pretraining and decoding strategies. Key findings show substantial IID-to-OOD performance drops, with generative models and multimodal pretraining offering advantages, while current automatic VQA metrics are overly stringent and human evaluation reveals additional correctness not captured automatically. The study calls for more rigorous, real-world-oriented evaluation protocols to measure robust VQA capabilities beyond benchmark-specific correlations.

Abstract

Vision-and-language (V&L) models pretrained on large-scale multimodal data have demonstrated strong performance on various tasks such as image captioning and visual question answering (VQA). The quality of such models is commonly assessed by measuring their performance on unseen data that typically comes from the same distribution as the training data. However, when evaluated under out-of-distribution (out-of-dataset) settings for VQA, we observe that these models exhibit poor generalization. We comprehensively evaluate two pretrained V&L models under different settings (i.e. classification and open-ended text generation) by conducting cross-dataset evaluations. We find that these models tend to learn to solve the benchmark, rather than learning the high-level skills required by the VQA task. We also find that in most cases generative models are less susceptible to shifts in data distribution compared to discriminative ones, and that multimodal pretraining is generally helpful for OOD generalization. Finally, we revisit assumptions underlying the use of automatic VQA evaluation metrics, and empirically show that their stringent nature repeatedly penalizes models for correct responses.

Figures (10)

• Figure 1: IID (highlighted in bold) vs. OOD performance. Top: ViLBERT pretrained on CC. Bottom: ALBEF pretrained on CC, VG, SBU, MS-COCO and C12M datasets. All models are initialized with BERT weights.
• Figure 2: IID (#) vs OOD performance when controlling for the shared shared answer set. Solid bars are as in \ref{['fig:iid_ood']}; stacked dotted bars are improvements when evaluating on questions with shared answer sets between IID and OOD settings. For IID, the shared answer set is computed with respect to a dataset denoted with *.
• Figure 3: Percentage point difference in VQA accuracy between models with and without multimodal pretraining, for OOD and IID (highlighted in bold) evaluations. All models are initialized with BERT weights.
• Figure 4: Difference in $\Delta\,\text{OOD}$ values between discriminative and generative models.
• Figure 5: Examples where models' prediction are correct but not accounted for in the ground-truth set. $\langle~\rangle$ denotes a list of unique (out of 10) ground-truth answers. VG ( VQAv2) model refers to a ViLBERT$_\text{DISC}$ fine-tuned on VG ( VQAv2).