Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Generation: Multi-Hop Reasoning for Factual Accuracy in Vision-Language Models

Shamima Hossain

TL;DR

The paper tackles factual inaccuracies in Visual Language Models by introducing a knowledge-guided, multi-hop reasoning framework that links visual perception to external knowledge graphs. The framework decomposes verification into hops: visual recognition, KG traversal, and fact-based caption refinement, and supports three knowledge representations (triple-based, hierarchical trees, bullet-points). Empirical results on a landmark-focused dataset show a substantial reduction in hallucinations—approximately thirty-two percent—along with improved factual consistency and interpretable reasoning traces. The work demonstrates the viability and potential of integrating external knowledge for reliable multimodal systems and sets the stage for scalable, knowledge-grounded captioning.

Abstract

Visual Language Models (VLMs) are powerful generative tools but often produce factually inaccurate outputs due to a lack of robust reasoning capabilities. While extensive research has been conducted on integrating external knowledge for reasoning in large language models (LLMs), such efforts remain underexplored in VLMs, where the challenge is compounded by the need to bridge multiple modalities seamlessly. This work introduces a framework for knowledge-guided reasoning in VLMs, leveraging structured knowledge graphs for multi-hop verification using image-captioning task to illustrate our framework. Our approach enables systematic reasoning across multiple steps, including visual entity recognition, knowledge graph traversal, and fact-based caption refinement. We evaluate the framework using hierarchical, triple-based and bullet-point based knowledge representations, analyzing their effectiveness in factual accuracy and logical inference. Empirical results show that our approach improves factual accuracy by approximately 31% on preliminary experiments on a curated dataset of mixtures from Google Landmarks v2, Conceptual captions and Coco captions revealing key insights into reasoning patterns and failure modes. This work demonstrates the potential of integrating external knowledge for advancing reasoning in VLMs, paving the way for more reliable and knowledgable multimodal systems.

Beyond Generation: Multi-Hop Reasoning for Factual Accuracy in Vision-Language Models

TL;DR

The paper tackles factual inaccuracies in Visual Language Models by introducing a knowledge-guided, multi-hop reasoning framework that links visual perception to external knowledge graphs. The framework decomposes verification into hops: visual recognition, KG traversal, and fact-based caption refinement, and supports three knowledge representations (triple-based, hierarchical trees, bullet-points). Empirical results on a landmark-focused dataset show a substantial reduction in hallucinations—approximately thirty-two percent—along with improved factual consistency and interpretable reasoning traces. The work demonstrates the viability and potential of integrating external knowledge for reliable multimodal systems and sets the stage for scalable, knowledge-grounded captioning.

Abstract

Visual Language Models (VLMs) are powerful generative tools but often produce factually inaccurate outputs due to a lack of robust reasoning capabilities. While extensive research has been conducted on integrating external knowledge for reasoning in large language models (LLMs), such efforts remain underexplored in VLMs, where the challenge is compounded by the need to bridge multiple modalities seamlessly. This work introduces a framework for knowledge-guided reasoning in VLMs, leveraging structured knowledge graphs for multi-hop verification using image-captioning task to illustrate our framework. Our approach enables systematic reasoning across multiple steps, including visual entity recognition, knowledge graph traversal, and fact-based caption refinement. We evaluate the framework using hierarchical, triple-based and bullet-point based knowledge representations, analyzing their effectiveness in factual accuracy and logical inference. Empirical results show that our approach improves factual accuracy by approximately 31% on preliminary experiments on a curated dataset of mixtures from Google Landmarks v2, Conceptual captions and Coco captions revealing key insights into reasoning patterns and failure modes. This work demonstrates the potential of integrating external knowledge for advancing reasoning in VLMs, paving the way for more reliable and knowledgable multimodal systems.

Paper Structure

This paper contains 19 sections, 4 equations, 2 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Studies
  3. Vision-Language Models for Image Captioning
  4. Studying hallucinations in image captioning task
  5. Knowledge Graph Integration Approaches
  6. Multi-Hop Reasoning Framework
  7. Exploring knowledge representation formats
  8. Triple-based Relations
  9. Hierarchical Knowledge Trees
  10. Bullet-point Facts
  11. Evaluation
  12. Experimental Setup
  13. Evaluation Metrics
  14. Appendix
  15. Relevant Knowledge Graph construction
  16. ...and 4 more sections

Figures (2)

  • Figure 1: A comparison of hallucinated entities in red and the factually correct entities after processing through our pipeline in green.
  • Figure 2: The system ingests an input image, generates a base caption using a VLM, and sequentially refines the caption through entity extraction, knowledge graph matching and augments the corrected entities to the prompt of the vlm to generate a factually accurate caption. Each module is color-coded and can operate independently, allowing for modular reasoning and analysis.