Knowledge-Aware Iterative Retrieval for Multi-Agent Systems

TL;DR

The paper tackles the challenge of open-domain, multi-hop question answering by introducing Knowledge-Aware Iterative Retrieval for Multi-Agent Systems, a framework that decouples external sources from an evolving internal knowledge cache to guide query generation and evidence selection. It employs an autonomous, agent-driven loop—Query Planning, Knowledge Update, and Contextual Filtering—to iteratively refine retrieval while balancing information diversity and factual accuracy, with dynamic termination based on unresolved gaps. A multi-agent extension explores competitive and collaborative configurations, showing two agents often deliver the best cost-performance on complex tasks, while three agents yield diminishing returns and higher costs. Across six open-domain QA benchmarks, the approach outperforms single-step baselines and conventional iterative retrieval methods in retrieval quality and downstream QA, demonstrating scalable, cost-efficient gains, especially for multi-hop reasoning. The work highlights practical implications for building scalable, transparent reasoning pipelines with verifiable intermediate steps and outlines avenues for broader validation and alternative collaboration models.

Abstract

We introduce a novel large language model (LLM)-driven agent framework, which iteratively refines queries and filters contextual evidence by leveraging dynamically evolving knowledge. A defining feature of the system is its decoupling of external sources from an internal knowledge cache that is progressively updated to guide both query generation and evidence selection. This design mitigates bias-reinforcement loops and enables dynamic, trackable search exploration paths, thereby optimizing the trade-off between exploring diverse information and maintaining accuracy through autonomous agent decision-making. Our approach is evaluated on a broad range of open-domain question answering benchmarks, including multi-step tasks that mirror real-world scenarios where integrating information from multiple sources is critical, especially given the vulnerabilities of LLMs that lack explicit reasoning or planning capabilities. The results show that the proposed system not only outperforms single-step baselines regardless of task difficulty but also, compared to conventional iterative retrieval methods, demonstrates pronounced advantages in complex tasks through precise evidence-based reasoning and enhanced efficiency. The proposed system supports both competitive and collaborative sharing of updated context, enabling multi-agent extension. The benefits of multi-agent configurations become especially prominent as task difficulty increases. The number of convergence steps scales with task difficulty, suggesting cost-effective scalability.

Figures (6)

• Figure 1: Overview of the single-agent architecture featuring a dynamic knowledge management and decision cycle. Iterative refinement balances accuracy and diversity through modular optimization.
• Figure 2: Multi-agent architectures: (Left) Collaborative model, where agents share and update a common knowledge repository. (Right) Competitive model, where agents maintain individual knowledge to reduce coordination overhead.
• Figure 3: Comparison of retrieval F1 across different datasets and agent configurations. The top sub-figure focuses on MuSiQue and 2WikiMultiHopQA, while the bottom sub-figure illustrates HotpotQA, TriviaQA, Natural Questions, and SQuAD.
• Figure 4: Answer F1 scores across retrieval systems and agent configurations. Multi-agent systems often yield higher F1 on complex datasets due to more precise context retrieval.
• Figure 5: Convergence behavior of the Knowledge-Aware Agent Retrieval Algorithm. The figure presents the number of retrieval iterations required across different agent configurations, evaluating the impact of iterative refinement on retrieval efficiency. The results highlight how agent collaboration influences convergence speed, demonstrating the effectiveness of knowledge-aware retrieval.