Visual Question Answering: A Survey of Methods and Datasets

TL;DR

This survey analyzes Visual Question Answering (VQA), a multimodal task that requires reasoning over images and natural language questions. It classifies VQA approaches into four main families: joint embeddings, attention, compositional neural modules, and knowledge-base–driven methods, detailing representative models and their strengths. It reviews a broad spectrum of datasets, from natural images to clipart and KB-enhanced sources, and highlights the Visual Genome scene graphs as a key but not fully sufficient annotation resource. The paper argues for richer external knowledge integration and continued NLP-informed enhancements to improve reasoning, interpretability, and generalization in VQA systems.

Abstract

Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. Given an image and a question in natural language, it requires reasoning over visual elements of the image and general knowledge to infer the correct answer. In the first part of this survey, we examine the state of the art by comparing modern approaches to the problem. We classify methods by their mechanism to connect the visual and textual modalities. In particular, we examine the common approach of combining convolutional and recurrent neural networks to map images and questions to a common feature space. We also discuss memory-augmented and modular architectures that interface with structured knowledge bases. In the second part of this survey, we review the datasets available for training and evaluating VQA systems. The various datatsets contain questions at different levels of complexity, which require different capabilities and types of reasoning. We examine in depth the question/answer pairs from the Visual Genome project, and evaluate the relevance of the structured annotations of images with scene graphs for VQA. Finally, we discuss promising future directions for the field, in particular the connection to structured knowledge bases and the use of natural language processing models.

Figures (8)

• Figure 1: (Top) A common approach to VQA is to map both the input image and question to a common embedding space (Section \ref{['subsec:Joint_e_app']}). These features are produced by deep convolutional and recurrent neural networks. They are combined in an output stage, which can take the form of a classifier (e.g. a multilayer perceptron) to predict short answers from predefined set or a recurrent network (e.g. an LSTM) to produce variable-length phrases. (Bottom) Attention mechanisms build up on this basic approach with a spatial selection of image features. Attention weights are derived from both the image and the question and allow the output stage to focus on relevant parts of the image.
• Figure 2: The Neural Module Networks (NMN, Section \ref{['subsubsec:modules']}) leverage the compositional structure of questions, e.g. here "Is there a red shape above a circle ?" from the Shapes dataset (Section \ref{['subsubsect:shapes']}). The parsing of the question leads to assembling modules that operate in the space of attentions. Two attend modules locate red shapes and circles, re-attend[above] shifts the attention above the circles, combine computes their intersection, and measure[is] inspects the final attention and determines that it is non-empty (figure adapted from andreas2015deep).
• Figure 3: Major datasets for VQA and their main characteristics. See Section \ref{['subsec:datasetreal']} for discussion.
• Figure 4: Example of a scene graph (structured annotation of an image) provided in the Visual Genome dataset krishnavisualgenome.
• Figure 5: We verify whether the answer to each question in the Visual Genome dataset can be found from the corresponding scene graph of the scene. Therefore, we first build the vocabulary of each image from the labels of all nodes and edges of its scene graph. Then, for each question, we check whether its answer can be found within the words or combination of words in the vocabulary of the corresponding image.