Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset

Aishwarya Agrawal, Aniruddha Kembhavi, Dhruv Batra, Devi Parikh

TL;DR

This work introduces Compositional VQA (C-VQA), a compositional split of the VQA v1.0 dataset in which test question-answer pairs are novel in composition relative to training data. It details a creation pipeline based on question concept extraction, grouping, and greedy re-splitting to maximize concept coverage in training while preventing QA leakage. The authors analyze distributions and benchmark several baselines, finding significant performance drops under C-VQA and shifts in model ranking, highlighting the gap in true compositional generalization. C-VQA provides a rigorous benchmark to assess and foster progress toward disentangled, compositional reasoning in VQA systems.

Abstract

Visual Question Answering (VQA) has received a lot of attention over the past couple of years. A number of deep learning models have been proposed for this task. However, it has been shown that these models are heavily driven by superficial correlations in the training data and lack compositionality -- the ability to answer questions about unseen compositions of seen concepts. This compositionality is desirable and central to intelligence. In this paper, we propose a new setting for Visual Question Answering where the test question-answer pairs are compositionally novel compared to training question-answer pairs. To facilitate developing models under this setting, we present a new compositional split of the VQA v1.0 dataset, which we call Compositional VQA (C-VQA). We analyze the distribution of questions and answers in the C-VQA splits. Finally, we evaluate several existing VQA models under this new setting and show that the performances of these models degrade by a significant amount compared to the original VQA setting.

C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset

TL;DR

This work introduces Compositional VQA (C-VQA), a compositional split of the VQA v1.0 dataset in which test question-answer pairs are novel in composition relative to training data. It details a creation pipeline based on question concept extraction, grouping, and greedy re-splitting to maximize concept coverage in training while preventing QA leakage. The authors analyze distributions and benchmark several baselines, finding significant performance drops under C-VQA and shifts in model ranking, highlighting the gap in true compositional generalization. C-VQA provides a rigorous benchmark to assess and foster progress toward disentangled, compositional reasoning in VQA systems.

Abstract

Visual Question Answering (VQA) has received a lot of attention over the past couple of years. A number of deep learning models have been proposed for this task. However, it has been shown that these models are heavily driven by superficial correlations in the training data and lack compositionality -- the ability to answer questions about unseen compositions of seen concepts. This compositionality is desirable and central to intelligence. In this paper, we propose a new setting for Visual Question Answering where the test question-answer pairs are compositionally novel compared to training question-answer pairs. To facilitate developing models under this setting, we present a new compositional split of the VQA v1.0 dataset, which we call Compositional VQA (C-VQA). We analyze the distribution of questions and answers in the C-VQA splits. Finally, we evaluate several existing VQA models under this new setting and show that the performances of these models degrade by a significant amount compared to the original VQA setting.

Paper Structure

This paper contains 7 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Compositional Visual Question Answering (C-VQA)
  4. C-VQA Creation
  5. C-VQA Analysis
  6. Baselines
  7. Conclusion

Figures (3)

  • Figure 1: Examples from our Compositional VQA (C-VQA) splits. Words belonging to same concepts are highlighted with same color to show the training instances from which the model can learn those concepts.
  • Figure 2: Distribution of questions by their first four words for a random sample of 60K questions for C-VQA train split (left) and C-VQA test split (right). The ordering of the words starts towards the center and radiates outwards. The arc length is proportional to the number of questions containing the word. White areas are words with contributions too small to show.
  • Figure 3: Distribution of answers per question type for a random sample of 60K questions for C-VQA train split (top) and C-VQA test split (bottom).