MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Framework

TL;DR

MMKB-RAG tackles the limitations of static parametric knowledge in multi-modal LLMs by introducing an endogenous, token-driven retrieval policy. It leverages the answer model's internal knowledge boundaries to generate RET, SRT, and MCT tokens that decide retrieval necessity, re-rank retrieved documents by model-driven relevance, and enforce consistency across sources. The framework achieves state-of-the-art performance on knowledge-based VQA benchmarks (e.g., E-VQA and InfoSeek) with notable gains on unseen and composite queries, and it demonstrates robustness on the M2KR dataset. By reducing irrelevant retrieval and enhancing factual accuracy, MMKB-RAG offers a practical path toward reliable, up-to-date multi-modal reasoning in real-world applications.

Abstract

Recent advancements in large language models (LLMs) and multi-modal LLMs have been remarkable. However, these models still rely solely on their parametric knowledge, which limits their ability to generate up-to-date information and increases the risk of producing erroneous content. Retrieval-Augmented Generation (RAG) partially mitigates these challenges by incorporating external data sources, yet the reliance on databases and retrieval systems can introduce irrelevant or inaccurate documents, ultimately undermining both performance and reasoning quality. In this paper, we propose Multi-Modal Knowledge-Based Retrieval-Augmented Generation (MMKB-RAG), a novel multi-modal RAG framework that leverages the inherent knowledge boundaries of models to dynamically generate semantic tags for the retrieval process. This strategy enables the joint filtering of retrieved documents, retaining only the most relevant and accurate references. Extensive experiments on knowledge-based visual question-answering tasks demonstrate the efficacy of our approach: on the E-VQA dataset, our method improves performance by +4.2% on the Single-Hop subset and +0.4% on the full dataset, while on the InfoSeek dataset, it achieves gains of +7.8% on the Unseen-Q subset, +8.2% on the Unseen-E subset, and +8.1% on the full dataset. These results highlight significant enhancements in both accuracy and robustness over the current state-of-the-art MLLM and RAG frameworks.

Figures (5)

• Figure 1: Examples illustrate that RAG may retrieve irrelevant and redundant documents, resulting in incorrect outcomes. Our approach leverages the MLLM's knowledge boundaries to filter references, retaining essential evidence for accurate answers.
• Figure 2: MMKB-RAG Pipeline. Given an input query, MMKB-RAG initially assesses retrieval necessity. For retrieval-dependent queries, we employ a two-stage filtering process (Steps 2 and 3) to ensure only high-quality reference documents are preserved. This curated context is then provided to the MLLM to generate comprehensive and accurate responses.
• Figure 3: Training strategy for token systems. RET determine whether to invoke the retrieval mechanism based on the model's response accuracy; SRT assess reference text quality by evaluating the model's performance when utilizing these documents as context; and CST uniformly evaluate all retrieved documents to deliver consistent reference documents.
• Figure 4: Sample qualitative results between MMKB-RAG and w/o MMKB-RAG.
• Figure 5: Illustration of our Token System, showcasing the effects of RET, SRT, and CST. Dashed lines indicate the operational positions of different tokens, with red text highlighting erroneous information and green text denoting correct information.