A Comprehensive Survey of Knowledge-Based Vision Question Answering Systems: The Lifecycle of Knowledge in Visual Reasoning Task

TL;DR

KB-VQA addresses the challenge of answering questions about images by harnessing external knowledge beyond the visual content. This survey provides a structured taxonomy that separates the field into knowledge representation, retrieval, and reasoning, while examining explicit, implicit, and hybrid knowledge sources and the role of LLMs in shaping these stages. It surveys diverse methods for query construction, knowledge alignment, graph-based representations, retrieval strategies, and reasoning paradigms, including attention-based fusion, graph neural networks, and Chain-of-Thought prompting. The work highlights persistent challenges such as noise, open-world retrieval, and the lack of unified benchmarks, and outlines future directions toward adaptive, human-aligned KB-VQA and unified evaluation across multi-source knowledge sources.

Abstract

Knowledge-based Vision Question Answering (KB-VQA) extends general Vision Question Answering (VQA) by not only requiring the understanding of visual and textual inputs but also extensive range of knowledge, enabling significant advancements across various real-world applications. KB-VQA introduces unique challenges, including the alignment of heterogeneous information from diverse modalities and sources, the retrieval of relevant knowledge from noisy or large-scale repositories, and the execution of complex reasoning to infer answers from the combined context. With the advancement of Large Language Models (LLMs), KB-VQA systems have also undergone a notable transformation, where LLMs serve as powerful knowledge repositories, retrieval-augmented generators and strong reasoners. Despite substantial progress, no comprehensive survey currently exists that systematically organizes and reviews the existing KB-VQA methods. This survey aims to fill this gap by establishing a structured taxonomy of KB-VQA approaches, and categorizing the systems into main stages: knowledge representation, knowledge retrieval, and knowledge reasoning. By exploring various knowledge integration techniques and identifying persistent challenges, this work also outlines promising future research directions, providing a foundation for advancing KB-VQA models and their applications.

Figures (5)

• Figure 1: A comparison between conventional VQA question and KB-VQA questions with different knowledge requirements.
• Figure 2: An overview of the generalized pipeline for knowledge-based visual question answering (KB-VQA), where each section provides an abstract representation of the main classifications. In this figure, "Section" is abbreviated as "Sec."; "Knowledge" is abbreviated as "Know." and "Chain-of-Thought" is abbreviated as "CoT".
• Figure 3: Taxonomy of Knowledge-based Vision Question Answering (KB-VQA) models, categorized into three key components: multimodal representation, knowledge retrieval, and knowledge reasoning. Relevant discussions are in Sections III, IV, and V. Model abbreviations follow their original papers. Methods highlighted by are apply with large language model (implicit knowledge) only, and others are with access to explicit knowledge.
• Figure 4: Two paradigm to fuse retrieved knowledge within the KB-VQA model. FiE concatenates all information first and processes all at once with the encoder. FiD processes the knowledge individually then concatenate before feeding them into the decoder.
• Figure 5: The representation model for various way of reasoning: designed attention-based, graph-based reasoning, chain-of-thought and dynamic (agent-based) reasoning. Please note the example are used for demonstration and more details about each model can refer to the original paper.