ReasonVQA: A Multi-hop Reasoning Benchmark with Structural Knowledge for Visual Question Answering

TL;DR

ReasonVQA introduces a scalable, knowledge-grounded VQA benchmark that requires multi-hop reasoning over external knowledge. The dataset links image objects to Wikidata via Visual Genome and Google Landmarks, and generates questions through a template bank with false choices, balancing, and careful train/test splits. Extensive baselines show ReasonVQA is substantially more challenging than existing VQA datasets, especially for 3-hop questions, while fine-tuning certain models yields notable gains. The work provides a generic, low-cost framework and open-source resources to expand knowledge-based VQA and drive robust multimodal reasoning research.

Abstract

In this paper, we propose a new dataset, ReasonVQA, for the Visual Question Answering (VQA) task. Our dataset is automatically integrated with structured encyclopedic knowledge and constructed using a low-cost framework, which is capable of generating complex, multi-hop questions. We evaluated state-of-the-art VQA models on ReasonVQA, and the empirical results demonstrate that ReasonVQA poses significant challenges to these models, highlighting its potential for benchmarking and advancing the field of VQA. Additionally, our dataset can be easily scaled with respect to input images; the current version surpasses the largest existing datasets requiring external knowledge by more than an order of magnitude.

Figures (16)

• Figure 1: Sample image and questions from ReasonVQA. Using an existing image, a question is formulated by reasoning through one or multiple hops over the knowledge graph. The generated questions span diverse domains.
• Figure 1: Example of linking an object from VG to a concept in Wikidata using Wordnet synset name. The Wikidata entity is retrieved by the WordNet synset ID, which is converted from the synset name using the NLTK package.
• Figure 2: Overview of the question generation process. The multiple choice question and answer are generated given the input image and the knowledge base. Our approach first utilizes the annotation from the dataset to link the object from the input image to a concept in knowledge base, then assembles the question from a predefined template, and finally generate the multiple choices.
• Figure 2: Example of linking a landmark from GLDv2 to a concept in Wikidata. The Wikidata entity is retrieved by its name, which is extracted from the Wikimedia URL in GLDv2.
• Figure 3: The process of question generation step-by-step from a Wikidata concept to the multiple choice question. The class hierarchy and all properties of the concept are queried and filtered down to a subset. Then each of the remaining properties is used to generate a multiple choice question.