RAGRouter-Bench: A Dataset and Benchmark for Adaptive RAG Routing

TL;DR

RAGRouter-Bench addresses the need for adaptive routing in Retrieval-Augmented Generation by introducing a large, multi-domain dataset and a five-paradigm benchmark that jointly evaluates generation quality and resource usage. It emphasizes query–corpus compatibility, proposing a dual-view analysis framework (corpus topology and embedding space, plus query type) to understand when different retrieval paradigms perform best. The study finds that no single paradigm dominates across all contexts and that corpus properties and query types jointly constrain effectiveness-efficiency trade-offs, with corpus fingerprints guiding routing decisions. This work provides a principled foundation for adaptive, interpretable, and scalable next-generation RAG systems and a benchmark to advance routing-aware RAG research.

Abstract

Retrieval-Augmented Generation (RAG) has become a core paradigm for grounding large language models with external knowledge. Despite extensive efforts exploring diverse retrieval strategies, existing studies predominantly focus on query-side complexity or isolated method improvements, lacking a systematic understanding of how RAG paradigms behave across different query-corpus contexts and effectiveness-efficiency trade-offs. In this work, we introduce RAGRouter-Bench, the first dataset and benchmark designed for adaptive RAG routing. RAGRouter-Bench revisits retrieval from a query-corpus compatibility perspective and standardizes five representative RAG paradigms for systematic evaluation across 7,727 queries and 21,460 documents spanning diverse domains. The benchmark incorporates three canonical query types together with fine-grained semantic and structural corpus metrics, as well as a unified evaluation for both generation quality and resource consumption. Experiments with DeepSeek-V3 and LLaMA-3.1-8B demonstrate that no single RAG paradigm is universally optimal, that paradigm applicability is strongly shaped by query-corpus interactions, and that increased advanced mechanism does not necessarily yield better effectiveness-efficiency trade-offs. These findings underscore the necessity of routing-aware evaluation and establish a foundation for adaptive, interpretable, and generalizable next-generation RAG systems.

Figures (17)

• Figure 1: Preliminary Study on Paradigm Conflict.Left: Accuracy of four RAG paradigms across two datasets and three query types. Right: Token consumption per paradigm on each dataset.
• Figure 2: Overview of the RAGRouter-Bench framework.Left: Query types with representative examples. Center: Five RAG paradigms as routing targets. Right: Multi-domain corpora with structural and semantic characterization. Bottom: Dual-axis evaluation covering response quality and resource efficiency.
• Figure 3: Overview of the five RAG paradigms evaluated in RAGRouter-Bench.Input: Query and corpus shared across all paradigms. Retrieval: Paradigm-specific pipelines differing in index structures and retrieval strategies. Generation: Retrieved context combined with query as prompt to LLM. Output: Final response.
• Figure 4: Query type taxonomy and dataset composition in RAGRouter-Bench.Left: Three query types definition. Right: Query type distribution across four datasets.
• Figure 5: Paradigm performance across datasets and query types. Each panel shows one RAG paradigm's LLM-as-a-Judge accuracy (Correct%), with rows as query types and columns as datasets. Asterisk (*) marks the best-performing paradigm for each combination.