Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning

TL;DR

This work tackles hallucinations and knowledge gaps in multimodal reasoning by introducing VaLiK, a fully annotation-free MMKG construction framework that converts images into image-specific textual descriptions using a cascade of Vision-Language Experts and refines them with cross-modal similarity verification. The purified descriptions are organized into MMKGs via LightRAG-guided prompting of an LLM, enabling effective retrieval and reasoning with cross-modal evidence. Experiments on CrisisMMD and ScienceQA demonstrate state-of-the-art multimodal reasoning performance while achieving substantial storage efficiency, highlighting VaLiK's potential for scalable, domain-adaptive knowledge grounding in LLMs.

Abstract

Multimodal reasoning in Large Language Models (LLMs) struggles with incomplete knowledge and hallucination artifacts, challenges that textual Knowledge Graphs (KGs) only partially mitigate due to their modality isolation. While Multimodal Knowledge Graphs (MMKGs) promise enhanced cross-modal understanding, their practical construction is impeded by semantic narrowness of manual text annotations and inherent noise in visual-semantic entity linkages. In this paper, we propose Vision-align-to-Language integrated Knowledge Graph (VaLiK), a novel approach for constructing MMKGs that enhances LLMs reasoning through cross-modal information supplementation. Specifically, we cascade pre-trained Vision-Language Models (VLMs) to align image features with text, transforming them into descriptions that encapsulate image-specific information. Furthermore, we developed a cross-modal similarity verification mechanism to quantify semantic consistency, effectively filtering out noise introduced during feature alignment. Even without manually annotated image captions, the refined descriptions alone suffice to construct the MMKG. Compared to conventional MMKGs construction paradigms, our approach achieves substantial storage efficiency gains while maintaining direct entity-to-image linkage capability. Experimental results on multimodal reasoning tasks demonstrate that LLMs augmented with VaLiK outperform previous state-of-the-art models. Our code is published at https://github.com/Wings-Of-Disaster/VaLiK.

Figures (7)

• Figure 1: (a) Training entity extraction models relies on extensive fine-grained annotations, increasing labeling costs. More examples are provided in Appendix B. (b) Capturing implicit semantic associations demands abstract comprehension or logical inference.
• Figure 2: Feature-aligned descriptions from VLMs introduce redundant and inaccurate relationship patterns.
• Figure 3: The pipeline of VaLiK: First, large-scale visual descriptions are generated using CoE-based VLMs. Then, a similarity verification mechanism is used to prune irrelevant information. Finally, MMKGs are constructed using LLMs based on LightRAG. The constructed MMKGs can assist LLMs in multimodal reasoning, alleviating the hallucination issues caused by incomplete knowledge.
• Figure 4: Impact analysis of VLM quantity on CrisisMMD.
• Figure 5: (a) The limited information contained in text-based KGs leads to inaccurate responses. (b) Leveraging MMKGs enables reasoning with enriched multimodal information to produce the correct answer.