Multimodal Reasoning with Multimodal Knowledge Graph

TL;DR

MR-MKG tackles hallucination and knowledge gaps in multimodal reasoning by integrating multimodal knowledge graphs with LLMs. It encodes MMKGs via RGAT, aligns modalities through a knowledge adapter and a cross-modal matching objective, and pretrains on an MMKG-grounded dataset, while freezing the LLM and visual encoder to keep parameter updates small. The approach achieves state-of-the-art results on ScienceQA and MARS with roughly 2.25% of the LLM’s parameters trained, highlighting strong gains in both multimodal QA and analogy reasoning. This work demonstrates that structured, cross-modal knowledge from MMKGs can significantly bolster reasoning efficiency and accuracy in large language models, with practical implications for robust multimodal AI systems.

Abstract

Multimodal reasoning with large language models (LLMs) often suffers from hallucinations and the presence of deficient or outdated knowledge within LLMs. Some approaches have sought to mitigate these issues by employing textual knowledge graphs, but their singular modality of knowledge limits comprehensive cross-modal understanding. In this paper, we propose the Multimodal Reasoning with Multimodal Knowledge Graph (MR-MKG) method, which leverages multimodal knowledge graphs (MMKGs) to learn rich and semantic knowledge across modalities, significantly enhancing the multimodal reasoning capabilities of LLMs. In particular, a relation graph attention network is utilized for encoding MMKGs and a cross-modal alignment module is designed for optimizing image-text alignment. A MMKG-grounded dataset is constructed to equip LLMs with initial expertise in multimodal reasoning through pretraining. Remarkably, MR-MKG achieves superior performance while training on only a small fraction of parameters, approximately 2.25% of the LLM's parameter size. Experimental results on multimodal question answering and multimodal analogy reasoning tasks demonstrate that our MR-MKG method outperforms previous state-of-the-art models.

Figures (9)

• Figure 1: (a) The inadequate knowledge encapsulated within textual KG results in the incorrect answer. (b) Our MR-MKG produces the correct answer by reasoning with richer multimodal information.
• Figure 2: The overview of our MR-MKG approach. Text, multimodal knowledge graph, and image are independently embedded and then concatenated to form prompt embedding tokens. A cross-modal alignment module is designed to enhance the image-text alignment through a matching task within MMKGs.
• Figure 3: Impact of numbers of knowledge triplets.
• Figure 4: Impact of numbers of KGE layers.
• Figure 5: Two examples from MRAS and scienceQA datasets. In case A, the model needs to predict coal based on an Analogical Example and the image of combustion. In case B, the model needs to select the correct answer based on the image and the question. Relevant entities for reasoning are marked in orange or highlighted with a red box.