mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs

TL;DR

mmRAG tackles the lack of modular, multi-modal evaluation for Retrieval-Augmented Generation (RAG) by constructing a unified corpus from six QA datasets spanning text, tables, and knowledge graphs, and by providing chunk-level and dataset-level relevance labels that enable direct assessment of retrieval and query routing in addition to generation. It introduces a three-phase construction (dataset collection, processing, annotation) and a comprehensive evaluation protocol with baselines for retrievers and generators, plus two routing strategies including an Oracle baseline; dataset-level relevance is defined as $S_{q, D} = \sum_{d \in D} \max_{c \in d} L_{q, c}$ to quantify each dataset's contribution. The results show that the BGE retriever often yields the strongest retrieval performance while LLM-based routers outperform semantic routers at small top-k, with generation quality correlating with retrieval quality and gaps remaining relative to the Oracle. The work is publicly released on Hugging Face under the Apache 2.0 license to spur modular, cross-modal RAG research and future extensions to include additional modalities and domain-specific benchmarks.

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing the capabilities of large language models. However, existing RAG evaluation predominantly focuses on text retrieval and relies on opaque, end-to-end assessments of generated outputs. To address these limitations, we introduce mmRAG, a modular benchmark designed for evaluating multi-modal RAG systems. Our benchmark integrates queries from six diverse question-answering datasets spanning text, tables, and knowledge graphs, which we uniformly convert into retrievable documents. To enable direct, granular evaluation of individual RAG components -- such as the accuracy of retrieval and query routing -- beyond end-to-end generation quality, we follow standard information retrieval procedures to annotate document relevance and derive dataset relevance. We establish baseline performance by evaluating a wide range of RAG implementations on mmRAG.

Figures (3)

• Figure 1: Construction of mmRAG.
• Figure 2: Evaluation of query routers (routing accuracy).
• Figure 3: Evaluation of query routers (generation quality).