Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GC-KBVQA: A New Four-Stage Framework for Enhancing Knowledge Based Visual Question Answering Performance

Mohammad Mahdi Moradi, Sudhir Mudur

TL;DR

This paper tackles knowledge-based visual question answering by addressing the inefficiency and irrelevance of auxiliary prompts in zero-shot setups. It introduces GC-KBVQA, a four-stage modular framework that grounds question-relevant image regions, generates and filters captions, creates caption-driven QA pairs, and composes a structured prompt for a pre-trained LLM to produce answers without task-specific training. The approach achieves strong zero-shot results on OK-VQA, A-OKVQA, and VQAv2, with notable efficiency and robustness across different LLM sizes, and ablations demonstrate the value of keyword-guided grounding, dual-caption generation, and content-aware filtering. The work emphasizes modular design and prompting as scalable, cost-effective strategies for knowledge-intensive VQA, while outlining limitations in temporal and numerical reasoning and proposing future directions to address them.

Abstract

Knowledge-Based Visual Question Answering (KB-VQA) methods focus on tasks that demand reasoning with information extending beyond the explicit content depicted in the image. Early methods relied on explicit knowledge bases to provide this auxiliary information. Recent approaches leverage Large Language Models (LLMs) as implicit knowledge sources. While KB-VQA methods have demonstrated promising results, their potential remains constrained as the auxiliary text provided may not be relevant to the question context, and may also include irrelevant information that could misguide the answer predictor. We introduce a novel four-stage framework called Grounding Caption-Guided Knowledge-Based Visual Question Answering (GC-KBVQA), which enables LLMs to effectively perform zero-shot VQA tasks without the need for end-to-end multimodal training. Innovations include grounding question-aware caption generation to move beyond generic descriptions and have compact, yet detailed and context-rich information. This is combined with knowledge from external sources to create highly informative prompts for the LLM. GC-KBVQA can address a variety of VQA tasks, and does not require task-specific fine-tuning, thus reducing both costs and deployment complexity by leveraging general-purpose, pre-trained LLMs. Comparison with competing KB-VQA methods shows significantly improved performance. Our code will be made public.

GC-KBVQA: A New Four-Stage Framework for Enhancing Knowledge Based Visual Question Answering Performance

TL;DR

This paper tackles knowledge-based visual question answering by addressing the inefficiency and irrelevance of auxiliary prompts in zero-shot setups. It introduces GC-KBVQA, a four-stage modular framework that grounds question-relevant image regions, generates and filters captions, creates caption-driven QA pairs, and composes a structured prompt for a pre-trained LLM to produce answers without task-specific training. The approach achieves strong zero-shot results on OK-VQA, A-OKVQA, and VQAv2, with notable efficiency and robustness across different LLM sizes, and ablations demonstrate the value of keyword-guided grounding, dual-caption generation, and content-aware filtering. The work emphasizes modular design and prompting as scalable, cost-effective strategies for knowledge-intensive VQA, while outlining limitations in temporal and numerical reasoning and proposing future directions to address them.

Abstract

Knowledge-Based Visual Question Answering (KB-VQA) methods focus on tasks that demand reasoning with information extending beyond the explicit content depicted in the image. Early methods relied on explicit knowledge bases to provide this auxiliary information. Recent approaches leverage Large Language Models (LLMs) as implicit knowledge sources. While KB-VQA methods have demonstrated promising results, their potential remains constrained as the auxiliary text provided may not be relevant to the question context, and may also include irrelevant information that could misguide the answer predictor. We introduce a novel four-stage framework called Grounding Caption-Guided Knowledge-Based Visual Question Answering (GC-KBVQA), which enables LLMs to effectively perform zero-shot VQA tasks without the need for end-to-end multimodal training. Innovations include grounding question-aware caption generation to move beyond generic descriptions and have compact, yet detailed and context-rich information. This is combined with knowledge from external sources to create highly informative prompts for the LLM. GC-KBVQA can address a variety of VQA tasks, and does not require task-specific fine-tuning, thus reducing both costs and deployment complexity by leveraging general-purpose, pre-trained LLMs. Comparison with competing KB-VQA methods shows significantly improved performance. Our code will be made public.

Paper Structure

This paper contains 18 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Keyword-Guided Visual Grounding
  5. Caption Generation and Filtering
  6. Caption-driven QA pairs
  7. Integrating Information for Answer Generation
  8. Experimental Evaluations
  9. Setup
  10. Main Results
  11. Ablation Study on LLM type and size
  12. Ablation Study on Grounding Dino Prompt Design
  13. Ablation Study on Use of Two Caption Generators
  14. Ablation Study on the number of Captions
  15. Ablation Study on Answer Generator Prompt Design
  16. ...and 3 more sections

Figures (1)

  • Figure 1: The architecture of GC-KBVQA leverages four pre-trained modules: (1) Question-relevant region extraction identifies and grounds the image regions pertinent to the question, (2) Pre-trained caption generator(s) generates captions for these grounded regions and applies filtering and ranking mechanisms to prioritize captions based on semantic relevance to the question, (3) Q&A pair generator constructs task-specific question-answer pairs derived from the selected captions, (4) Answer generator integrates captions, and QA pairs to deliver contextually coherent answers.