QIRL: Boosting Visual Question Answering via Optimized Question-Image Relation Learning

TL;DR

A novel framework, Optimized Question-Image Relation Learning (QIRL), which employs a generation-based self-supervised learning strategy and can be integrated with various VQA models, and achieves state-of-the-art results.

Abstract

Existing debiasing approaches in Visual Question Answering (VQA) primarily focus on enhancing visual learning, integrating auxiliary models, or employing data augmentation strategies. However, these methods exhibit two major drawbacks. First, current debiasing techniques fail to capture the superior relation between images and texts because prevalent learning frameworks do not enable models to extract deeper correlations from highly contrasting samples. Second, they do not assess the relevance between the input question and image during inference, as no prior work has examined the degree of input relevance in debiasing studies. Motivated by these limitations, we propose a novel framework, Optimized Question-Image Relation Learning (QIRL), which employs a generation-based self-supervised learning strategy. Specifically, two modules are introduced to address the aforementioned issues. The Negative Image Generation (NIG) module automatically produces highly irrelevant question-image pairs during training to enhance correlation learning, while the Irrelevant Sample Identification (ISI) module improves model robustness by detecting and filtering irrelevant inputs, thereby reducing prediction errors. Furthermore, to validate our concept of reducing output errors through filtering unrelated question-image inputs, we propose a specialized metric to evaluate the performance of the ISI module. Notably, our approach is model-agnostic and can be integrated with various VQA models. Extensive experiments on VQA-CPv2 and VQA-v2 demonstrate the effectiveness and generalization ability of our method. Among data augmentation strategies, our approach achieves state-of-the-art results.

Figures (12)

• Figure 1: Illustration of Language Bias. VQA models often exploit superficial correlations between questions and answers, neglecting the image content during inference.
• Figure 2: Illustration of Existing Approach for Generating Irrelevant QI Pairs. Although designed to produce irrelevant pairs, this method is flawed: as shown in the bottom row, both images contain a cat while the question pertains to a cat, rendering the generated pair relevant.
• Figure 3: Influence of Relevant and Irrelevant QI Pairs on Predictions. The top instance represents a relevant QI pair, whereas the bottom instance represents an irrelevant QI pair. The red ellipse denotes visual information, and the blue ellipse denotes semantic information. Note that the information extraction and analysis depicted here is schematic and does not reflect the model's actual operation.
• Figure 4: Proposed QIRL Architecture Comprises Three Components: a VQA model, a Generation of QI Pairs module, and a QI Correlation Learning module. The VQA model produces predictions based on the inputs; the Generation of QI Pairs process (purple zone) enhances the quality of generated QI pairs by ensuring they are highly irrelevant; and the QI Correlation Learning process (aqua zone) improves VQA performance by alleviating language bias and increasing model capacity.
• Figure 5: Sentence Revision Process. (a) depicts a parse tree for phrase detection; (b) illustrates two operations used to generate candidate sentences, where the blue words and the red words represent the Substitution operation and the Removal operation, respectively.