Robust Visual Question Answering: Datasets, Methods, and Future Challenges

TL;DR

The paper addresses the robustness of Visual Question Answering (VQA) systems by critically examining biases that enable memorization over true grounding. It surveys datasets (ID and OOD), evaluation metrics, and debiasing methods (ensemble learning, data augmentation, self-supervised contrastive learning, and answer re-ranking), and assesses the robustness of vision-language pre-training models on VQA tasks. Key contributions include a taxonomy of debiasing techniques, an analysis of VLM robustness in both discriminative and generative VQA settings, and a discussion of future directions for improving annotation quality, dataset design, and evaluation protocols. The study highlights that current methods often trade off ID performance for improved OOD generalization and emphasizes the need for multi-dataset robustness assessment and integration of debiasing with large-scale VLM pre-training to impact real-world VQA deployment.

Abstract

Visual question answering requires a system to provide an accurate natural language answer given an image and a natural language question. However, it is widely recognized that previous generic VQA methods often exhibit a tendency to memorize biases present in the training data rather than learning proper behaviors, such as grounding images before predicting answers. Therefore, these methods usually achieve high in-distribution but poor out-of-distribution performance. In recent years, various datasets and debiasing methods have been proposed to evaluate and enhance the VQA robustness, respectively. This paper provides the first comprehensive survey focused on this emerging fashion. Specifically, we first provide an overview of the development process of datasets from in-distribution and out-of-distribution perspectives. Then, we examine the evaluation metrics employed by these datasets. Thirdly, we propose a typology that presents the development process, similarities and differences, robustness comparison, and technical features of existing debiasing methods. Furthermore, we analyze and discuss the robustness of representative vision-and-language pre-training models on VQA. Finally, through a thorough review of the available literature and experimental analysis, we discuss the key areas for future research from various viewpoints.

Figures (14)

• Figure 1: Illustration of generic VQA methods in the in-distribution and out-of-distribution test scenarios. They predict answers by learning strong language bias, such as the connections between critical words "what", and "sports" in questions and the most frequent answer "tennis", rather than grounding images, which results in their high In-Distribution (ID) but poor Out-Of-Distribution (OOD) test performance. The ID situation refers to scenarios where the distribution is similar to that of the corresponding training split. The OOD scenario, on the other hand, applies to cases where the distribution differs or even opposes that of the training split.
• Figure 2: VQA v1 example. Each question in this dataset is annotated by ten humans. Therefore, a question may have multiple different ground-truth answers. The dataset is classified into three categories according to answer types: "Number", "Yes/No", and "Other".
• Figure 3: Illustration of balancing answer distributions in VQA v2. This dataset incorporates a complementary image that pertains to the same question in VQA v1 but has a different answer.
• Figure 4: GQA example. This dataset provides a scene graph for each image and a functional program for each question. The program enumerates a series of logical operations (reasoning steps) necessary to obtain the answer.
• Figure 5: Answer distributions under specific types of questions in VQA-CP v1 training and test splits.