Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey

Jiayi Kuang, Jingyou Xie, Haohao Luo, Ronghao Li, Zhe Xu, Xianfeng Cheng, Yinghui Li, Xika Lin, Ying Shen

TL;DR

This survey gives an up-to-date synthesis of natural language understanding of images and text, as well as the knowledge reasoning module based on image-question information on the core VQA tasks.

Abstract

Visual Question Answering (VQA) is a challenge task that combines natural language processing and computer vision techniques and gradually becomes a benchmark test task in multimodal large language models (MLLMs). The goal of our survey is to provide an overview of the development of VQA and a detailed description of the latest models with high timeliness. This survey gives an up-to-date synthesis of natural language understanding of images and text, as well as the knowledge reasoning module based on image-question information on the core VQA tasks. In addition, we elaborate on recent advances in extracting and fusing modal information with vision-language pretraining models and multimodal large language models in VQA. We also exhaustively review the progress of knowledge reasoning in VQA by detailing the extraction of internal knowledge and the introduction of external knowledge. Finally, we present the datasets of VQA and different evaluation metrics and discuss possible directions for future work.

Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey

TL;DR

This survey gives an up-to-date synthesis of natural language understanding of images and text, as well as the knowledge reasoning module based on image-question information on the core VQA tasks.

Abstract

Visual Question Answering (VQA) is a challenge task that combines natural language processing and computer vision techniques and gradually becomes a benchmark test task in multimodal large language models (MLLMs). The goal of our survey is to provide an overview of the development of VQA and a detailed description of the latest models with high timeliness. This survey gives an up-to-date synthesis of natural language understanding of images and text, as well as the knowledge reasoning module based on image-question information on the core VQA tasks. In addition, we elaborate on recent advances in extracting and fusing modal information with vision-language pretraining models and multimodal large language models in VQA. We also exhaustively review the progress of knowledge reasoning in VQA by detailing the extraction of internal knowledge and the introduction of external knowledge. Finally, we present the datasets of VQA and different evaluation metrics and discuss possible directions for future work.

Paper Structure

This paper contains 73 sections, 28 equations, 9 figures, 6 tables.

Table of Contents

  1. INTRODUCTION
  2. What is Visual Question Answering?
  3. Comparison with Other Survey Works
  4. Contribution of this Survey
  5. COMPREHENSION OF IMAGE AND TEXT IN VQA
  6. Feature Extraction
  7. Visual Feature Extraction
  8. Textual Feature Extraction
  9. Improvement of Embedding
  10. Fusion Mechanism
  11. Simple Vector Operations
  12. Deep Learning Method
  13. Non-Attention Based Deep Learning
  14. Attention Based Deep Learning
  15. GNN-based Fusion Approaches
  16. ...and 58 more sections

Figures (9)

  • Figure 1: The data used in VQA tasks, with the conclusion of the understanding and inference methods from conventional models to multimodal large language models.
  • Figure 2: Taxonomy Graph of VQA Task.
  • Figure 3: Percentage distribution of the usage of visual and textual feature extractors.
  • Figure 4: Comparison of fusion mechanism utilizing (a) non-attention based deep learning and (b) attention based deep learning.
  • Figure 5: Comparison of fusion mechanism utilizing (a) dual-stream pre-trained models and (b) single-stream pre-trained models.
  • ...and 4 more figures