Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Find The Gap: Knowledge Base Reasoning For Visual Question Answering

Elham J. Barezi, Parisa Kordjamshidi

TL;DR

The paper addresses KB-VQA by proposing a supervised retrieval framework that extracts relevant external KB triplets and scene-graph facts, followed by two reasoning backbones: a task-specific neural architecture and a large language model. It demonstrates that explicit retrieval substantially boosts VQA performance and that LLMs excel at single-hop reasoning but struggle with multi-hop scenarios compared to fine-tuned neural networks, while still benefiting from implicit knowledge for KB-related questions. The study highlights the complementary strengths of explicit knowledge retrieval and implicit LLM knowledge, showing that external KB remains valuable even with strong LLMs, and underscores the importance of designing robust, multi-hop reasoning modules. Practical implications include improved KB-VQA systems that combine supervised retrieval with structured reasoning to mitigate noise and enhance interpretability. The work also points to future avenues such as dynamic retrieval, better high-level visual representations, and broader evaluation across datasets.

Abstract

We analyze knowledge-based visual question answering, for which given a question, the models need to ground it into the visual modality and retrieve the relevant knowledge from a given large knowledge base (KB) to be able to answer. Our analysis has two folds, one based on designing neural architectures and training them from scratch, and another based on large pre-trained language models (LLMs). Our research questions are: 1) Can we effectively augment models by explicit supervised retrieval of the relevant KB information to solve the KB-VQA problem? 2) How do task-specific and LLM-based models perform in the integration of visual and external knowledge, and multi-hop reasoning over both sources of information? 3) Is the implicit knowledge of LLMs sufficient for KB-VQA and to what extent it can replace the explicit KB? Our results demonstrate the positive impact of empowering task-specific and LLM models with supervised external and visual knowledge retrieval models. Our findings show that though LLMs are stronger in 1-hop reasoning, they suffer in 2-hop reasoning in comparison with our fine-tuned NN model even if the relevant information from both modalities is available to the model. Moreover, we observed that LLM models outperform the NN model for KB-related questions which confirms the effectiveness of implicit knowledge in LLMs however, they do not alleviate the need for external KB.

Find The Gap: Knowledge Base Reasoning For Visual Question Answering

TL;DR

The paper addresses KB-VQA by proposing a supervised retrieval framework that extracts relevant external KB triplets and scene-graph facts, followed by two reasoning backbones: a task-specific neural architecture and a large language model. It demonstrates that explicit retrieval substantially boosts VQA performance and that LLMs excel at single-hop reasoning but struggle with multi-hop scenarios compared to fine-tuned neural networks, while still benefiting from implicit knowledge for KB-related questions. The study highlights the complementary strengths of explicit knowledge retrieval and implicit LLM knowledge, showing that external KB remains valuable even with strong LLMs, and underscores the importance of designing robust, multi-hop reasoning modules. Practical implications include improved KB-VQA systems that combine supervised retrieval with structured reasoning to mitigate noise and enhance interpretability. The work also points to future avenues such as dynamic retrieval, better high-level visual representations, and broader evaluation across datasets.

Abstract

We analyze knowledge-based visual question answering, for which given a question, the models need to ground it into the visual modality and retrieve the relevant knowledge from a given large knowledge base (KB) to be able to answer. Our analysis has two folds, one based on designing neural architectures and training them from scratch, and another based on large pre-trained language models (LLMs). Our research questions are: 1) Can we effectively augment models by explicit supervised retrieval of the relevant KB information to solve the KB-VQA problem? 2) How do task-specific and LLM-based models perform in the integration of visual and external knowledge, and multi-hop reasoning over both sources of information? 3) Is the implicit knowledge of LLMs sufficient for KB-VQA and to what extent it can replace the explicit KB? Our results demonstrate the positive impact of empowering task-specific and LLM models with supervised external and visual knowledge retrieval models. Our findings show that though LLMs are stronger in 1-hop reasoning, they suffer in 2-hop reasoning in comparison with our fine-tuned NN model even if the relevant information from both modalities is available to the model. Moreover, we observed that LLM models outperform the NN model for KB-related questions which confirms the effectiveness of implicit knowledge in LLMs however, they do not alleviate the need for external KB.
Paper Structure (20 sections, 2 equations, 4 figures, 4 tables)

This paper contains 20 sections, 2 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Plain VQA methods
  4. KB processing methods
  5. Large Language Models as Implicit Knowledge Sources
  6. The Proposed Method
  7. External Knowledge Retrieval
  8. Integration and Reasoning Using Task-specific Neural Architectures
  9. Integration and Reasoning using Large Language Models
  10. Experimental Results
  11. Datasets
  12. Implementation Details
  13. Retrieval Result
  14. Result and Discussion
  15. Question-type Analysis
  16. ...and 5 more sections

Figures (4)

  • Figure 1: The Diagram of our supervised retrieval model. despite previous methods, we added some layers on top of frozen Roberta to adjust the Roberta embeddings to knowledge retrieval task using supporting knowledge presented in the ground-truth reason.
  • Figure 2: An example krvqa dataset. given image, question, answer, and Supporting reason. The green part of the reason is extracted from the image, and the red part from the external KB.
  • Figure 3: the architecture of our proposed task-specific neural architecture. The attention modules iteratively update a query to solve multi-hop reasoning. The integration layer combines external and visual knowledge to provide the final output using a classifier layer. We initialize the query with question embedding and use KB and SG triplets as keys and values. The dashed arrows update the query iteratively using the output of attention-modules.
  • Figure 4: Architecture of our LLM model. The relevant knowledge is given as a part of the prompt