A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces

TL;DR

This paper addresses scaling retrieval-augmented generation by enabling agentic control over retrieval. It introduces A-RAG, an agentic RAG framework with hierarchical retrieval interfaces (keyword search, semantic search, chunk read) that allow LLMs to autonomously decide how to search and gather evidence. Across four multi-hop QA benchmarks, A-RAG or its naive variant outperforms prior Graph-RAG and Workflow RAG methods, while maintaining efficient context usage. The work demonstrates favorable test-time scaling with model and compute, and releases code to facilitate further research into agent-driven retrieval systems.

Abstract

Frontier language models have demonstrated strong reasoning and long-horizon tool-use capabilities. However, existing RAG systems fail to leverage these capabilities. They still rely on two paradigms: (1) designing an algorithm that retrieves passages in a single shot and concatenates them into the model's input, or (2) predefining a workflow and prompting the model to execute it step-by-step. Neither paradigm allows the model to participate in retrieval decisions, preventing efficient scaling with model improvements. In this paper, we introduce A-RAG, an Agentic RAG framework that exposes hierarchical retrieval interfaces directly to the model. A-RAG provides three retrieval tools: keyword search, semantic search, and chunk read, enabling the agent to adaptively search and retrieve information across multiple granularities. Experiments on multiple open-domain QA benchmarks show that A-RAG consistently outperforms existing approaches with comparable or lower retrieved tokens, demonstrating that A-RAG effectively leverages model capabilities and dynamically adapts to different RAG tasks. We further systematically study how A-RAG scales with model size and test-time compute. We will release our code and evaluation suite to facilitate future research. Code and evaluation suite are available at https://github.com/Ayanami0730/arag.

Figures (8)

• Figure 1: Two paradigms comparison and performance results.
• Figure 2: Comparison of three paradigms. We identify three principles of agentic autonomy: Autonomous Strategy, Iterative Execution, and Interleaved Tool Use. Only A-RAG satisfies all three, making it a truly agentic framework.
• Figure 3: Overview of A-RAG framework. The agent iteratively uses hierarchical retrieval tools (keyword search, semantic search, chunk read) to gather information from the corpus and autonomously decides when to provide the final answer.
• Figure 4: Test-time scaling analysis on MuSiQue-300. Left two: LLM-Acc vs. max steps with GPT-5-mini and GPT-4o-mini. Right two: LLM-Acc vs. reasoning effort with GPT-5-mini and GPT-5.
• Figure 5: Failure mode distribution of A-RAG. Top: primary categories. Bottom: breakdown of reasoning chain errors.