RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval Augmented Generation Systems

TL;DR

RAGCap-Bench targets the gap in evaluating intermediate steps of agentic RAG systems by introducing a capability-oriented MCQ benchmark across Planning, Evidence Extraction, Grounded Reasoning, and Noise Robustness. It constructs 255 questions from open benchmarks using Vanilla and Error-Guided MCQ generation and expert annotations, and quantifies performance with EM and macro-F1 per capability. Results show that slow-thinking LLMs with stronger RAGCap perform better end-to-end, that informative prompts improve intermediate task performance, and that RAGCap-Bench scores correlate with downstream QA outcomes. The findings suggest that improving intermediate capabilities and employing LLM-based evaluation of intermediate outputs can meaningfully enhance agentic RAG systems in open-web, dynamic settings.

Abstract

Retrieval-Augmented Generation (RAG) mitigates key limitations of Large Language Models (LLMs)-such as factual errors, outdated knowledge, and hallucinations-by dynamically retrieving external information. Recent work extends this paradigm through agentic RAG systems, where LLMs act as agents to iteratively plan, retrieve, and reason over complex queries. However, these systems still struggle with challenging multi-hop questions, and their intermediate reasoning capabilities remain underexplored. To address this, we propose RAGCap-Bench, a capability-oriented benchmark for fine-grained evaluation of intermediate tasks in agentic RAG workflows. We analyze outputs from state-of-the-art systems to identify common tasks and the core capabilities required for their execution, then construct a taxonomy of typical LLM errors to design targeted evaluation questions. Experiments show that "slow-thinking" models with stronger RAGCap performance achieve better end-to-end results, underscoring the benchmark's validity and the importance of enhancing these intermediate capabilities.

Figures (18)

• Figure 1: Left: Proportion of samples in each MCQ type. Right: Average number of options in each MCQ type.
• Figure 2: General agentic RAG pipeline and data construction processes for RAGCap-Bench. We take queries from open-source question-answering (QA) datasets. We then run different deep research systems and collect the intermediate outputs. Two main strategies, Vanilla Generation and Error-Guided Generation are deployed to generate the MCQs based on queries and the intermediate outputs. The generated MCQs are filtered to ensure the quality and experts are recruited to provide the answers.
• Figure 3: RAGCap-Bench overall EM scores for different fast-thinking (left) and slow-thinking models (right), with informative (orange) and bare (blue) prompts.
• Figure 4: Correlation of performance on RAGCap-Bench with performance on InfoDeepSeek and BrowseComp-Zh, for Qwen3-8B, Qwen3-32B and Qwen3-235B.
• Figure 5: Convergent (top) and divergent (bottom) planning capabilities required for different user queries.