MiRAGE: A Multiagent Framework for Generating Multimodal Multihop Question-Answer Dataset for RAG Evaluation

TL;DR

The paper tackles the need for domain-specific, multimodal, multi-hop evaluation benchmarks for retrieval-augmented generation systems. It introduces MiRAGE, a swarm-based multi-agent framework that builds semantic, multimodal context, generates expert-level QA pairs across four domains, and enforces factual grounding via an adversarial verifier and domain persona injection. By moving beyond linear synthetic pipelines to recursive context expansion and hierarchical deduplication, MiRAGE produces datasets with higher reasoning complexity while preserving domain structure. Empirical results across finance, regulation, science, and journalism show strong faithfulness and meaningful hops, though visual grounding remains a challenge. The work establishes a robust, automated benchmark-generation approach that can stress-test next-generation RAG systems on proprietary, multimodal corpora.

Abstract

The rapid evolution of Retrieval-Augmented Generation (RAG) toward multimodal, high-stakes enterprise applications has outpaced the development of domain specific evaluation benchmarks. Existing datasets often rely on general-domain corpora or purely textual retrieval, failing to capture the complexity of specialized technical documents where information is inextricably multimodal and reasoning requires synthesizing disjoint evidence. We address this gap by introducing MiRAGE, a Multiagent framework for RAG systems Evaluation, that leverages a collaborative swarm of specialized agents to generate verified, domain-specific, multimodal, and multi-hop Question-Answer datasets. MiRAGE orchestrates a swarm of specialized agents: a recursive context optimization loop to aggregate scattered evidence, an adversarial verifier agent to guarantee factual grounding, and an agent to recognize the expert persona and the relevant domain to mimic expert cognitive workflows. Extensive empirical evaluation across four distinct domains (regulations, finance, quantitative biology, and journalism) demonstrates that MiRAGE generates datasets with significantly higher reasoning complexity (>2.3 average hops) and factual faithfulness. Our ablation studies point that MiRAGE can be powered by LLMs if textual descriptions of the images are available. Visual grounding still remains a frontier. By automating the creation of gold standard evaluation datasets that reflect the latent thematic structure of proprietary corpora, MiRAGE provides the necessary infrastructure to rigorously benchmark the next generation information retrieval systems.

Figures (2)

• Figure 1: The multiagent framework of MiRAGE to evaluate RAG systems
• Figure 2: A sample question-answer pair generated from two related chunks with relevant keywords highlighted