Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering

Zhiyang Li, Ao Ke, Yukun Cao, Xike Xie

TL;DR

KG-ViP is proposed, a unified framework that empowers MLLMs by fusing scene graphs and commonsense graphs, synthesizing a unified structured context that facilitates reliable multi-modal reasoning.

Abstract

Multi-modal Large Language Models (MLLMs) for Visual Question Answering (VQA) often suffer from dual limitations: knowledge hallucination and insufficient fine-grained visual perception. Crucially, we identify that commonsense graphs and scene graphs provide precisely complementary solutions to these respective deficiencies by providing rich external knowledge and capturing fine-grained visual details. However, prior works typically treat them in isolation, overlooking their synergistic potential. To bridge this gap, we propose KG-ViP, a unified framework that empowers MLLMs by fusing scene graphs and commonsense graphs. The core of the KG-ViP framework is a novel retrieval-and-fusion pipeline that utilizes the query as a semantic bridge to progressively integrate both graphs, synthesizing a unified structured context that facilitates reliable multi-modal reasoning. Extensive experiments on FVQA 2.0+ and MVQA benchmarks demonstrate that KG-ViP significantly outperforms existing VQA methods.

KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering

TL;DR

KG-ViP is proposed, a unified framework that empowers MLLMs by fusing scene graphs and commonsense graphs, synthesizing a unified structured context that facilitates reliable multi-modal reasoning.

Abstract

Multi-modal Large Language Models (MLLMs) for Visual Question Answering (VQA) often suffer from dual limitations: knowledge hallucination and insufficient fine-grained visual perception. Crucially, we identify that commonsense graphs and scene graphs provide precisely complementary solutions to these respective deficiencies by providing rich external knowledge and capturing fine-grained visual details. However, prior works typically treat them in isolation, overlooking their synergistic potential. To bridge this gap, we propose KG-ViP, a unified framework that empowers MLLMs by fusing scene graphs and commonsense graphs. The core of the KG-ViP framework is a novel retrieval-and-fusion pipeline that utilizes the query as a semantic bridge to progressively integrate both graphs, synthesizing a unified structured context that facilitates reliable multi-modal reasoning. Extensive experiments on FVQA 2.0+ and MVQA benchmarks demonstrate that KG-ViP significantly outperforms existing VQA methods.
Paper Structure (27 sections, 1 equation, 4 figures, 6 tables)

This paper contains 27 sections, 1 equation, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. MLLMs for Knowledge-based VQA
  4. Scene Graph for MLLM Reasoning
  5. Methodology
  6. Multi-modal Graph Construction
  7. Commonsense Graph Construction
  8. Scene Graph Generation
  9. Two-stage Retrieval
  10. Graph Fusion and Refinement
  11. Evaluation
  12. Benchmarks
  13. Experimental Setup
  14. Main Results
  15. Ablations and Analysis
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Complementary roles of Scene Graphs and Commonsense Graphs.Using a scene from Inception as an example: Given the query asking "Who is the man holding the cup?", the scene graph explicitly captures visual relations (e.g., holding) yet fails to recognize character identities (Cobb, Robert). Conversely, the commonsense graph offers semantic facts (e.g., Cobb targets Robert) but suffers from grounding ambiguity, as it lacks visual cues to anchor abstract nodes to the specific person in the image. Thus, neither graph alone can support the complete reasoning chain.
  • Figure 2: The Overview of KG-ViP Framework.KG-ViP operates in three stages. First, we prepare a domain Commonsense Graph$\mathcal{G}_c$ and derive a Scene Graph$\mathcal{G}_s$ from the input image $I_q$. Second, the query $Q$ retrieves a text-guided subgraph $\mathcal{G}_{c,T}$ from $\mathcal{G}_c$, which is used to filter $\mathcal{G}_s$ into a refined $\hat{\mathcal{G}}_s$. Subsequently, $\hat{\mathcal{G}}_s$ guides the retrieval of visual knowledge $\mathcal{G}_{c,I}$ to form the final commonsense subgraph $\tilde{\mathcal{G}}_c$. Finally, $\hat{\mathcal{G}}_s$ and $\tilde{\mathcal{G}}_c$ are fused into a unified graph $\mathcal{G}_f$, providing the structured context for the MLLM to generate the answer $A$.
  • Figure 3: Impact of vision encoder selection on KG-ViP.
  • Figure 4: Qualitative comparison of VQA results. The visualization demonstrates the reasoning process and answer generation across different methods.