From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities

TL;DR

This work presents a survey in the domain of VQA that delves into the intricacies of V QA datasets and methods over the field's history, introduces a detailed taxonomy to categorize the facets of VZA, and highlights the recent trends, challenges, and scopes for improvement.

Abstract

The multimodal task of Visual Question Answering (VQA) encompassing elements of Computer Vision (CV) and Natural Language Processing (NLP), aims to generate answers to questions on any visual input. Over time, the scope of VQA has expanded from datasets focusing on an extensive collection of natural images to datasets featuring synthetic images, video, 3D environments, and various other visual inputs. The emergence of large pre-trained networks has shifted the early VQA approaches relying on feature extraction and fusion schemes to vision language pre-training (VLP) techniques. However, there is a lack of comprehensive surveys that encompass both traditional VQA architectures and contemporary VLP-based methods. Furthermore, the VLP challenges in the lens of VQA haven't been thoroughly explored, leaving room for potential open problems to emerge. Our work presents a survey in the domain of VQA that delves into the intricacies of VQA datasets and methods over the field's history, introduces a detailed taxonomy to categorize the facets of VQA, and highlights the recent trends, challenges, and scopes for improvement. We further generalize VQA to multimodal question answering, explore tasks related to VQA, and present a set of open problems for future investigation. The work aims to navigate both beginners and experts by shedding light on the potential avenues of research and expanding the boundaries of the field.

Figures (8)

• Figure 1: Taxonomy and definitions of several VQA tasks based on domain (where and how VQA is applied), the modality (type and source of information), and answer generation (type of output and how is produced).
• Figure 2: Overview of a visually impaired assistance system similar to the VizWiz Mobile App bigham2010vizwiz utilizing Multimodal Large Language Models (MLLMs). The app allows visually impaired users to take pictures of their environment along with a voice-recorded question. Automatic Speech Recognition (ASR) processes the audio input, generating a textual question that is sent to the MLLM along with the image. The MLLM then produces a comprehensive answer, which is converted back to audio through Speech Synthesis. Alternatively, the audio can be sent directly to the MLLM for Audio VQA, and dedicated modules for ASR, Speech Synthesis, and VQA can also be used separately.
• Figure 3: A standard visual dialog system should be capable of a multitude of unimodal and bimodal tasks. Some of the tasks include but are not limited to tracking conversational history during question answering, performing any form of textual or visual reasoning, being able to answer questions that require external knowledge, generating image captions, and generating synthetic images.
• Figure 4: Timeline of popular VQA datasets
• Figure 5: Overview of the traditional pre-transformer VQA architecture based on Joint Embedding and Attention. CNN-RNN-based encoder pairs, Multimodal Fusion, and classification head were used.