FAIR-RAG: Faithful Adaptive Iterative Refinement for Retrieval-Augmented Generation

TL;DR

FAIR-RAG introduces a faithful, adaptive, iterative refinement framework for retrieval-augmented generation. Central to the approach is the Structured Evidence Assessment (SEA), which deconstructs queries into a checklist and identifies explicit information gaps, driving targeted adaptive query refinement across iterative cycles. The system alternates between evidence gathering (hybrid retrieval and filtering) and rigorous verification before generation, ensuring outputs are grounded in verifiable sources. Empirical results on HotpotQA, 2WikiMultiHopQA, MusiQue, and TriviaQA demonstrate state-of-the-art performance on complex multi-hop tasks and strong results on single-hop questions, with clear evidence that iterative refinement and adaptive resource allocation yield substantial gains while maintaining faithfulness.

Abstract

While Retrieval-Augmented Generation (RAG) mitigates hallucination and knowledge staleness in Large Language Models (LLMs), existing frameworks often falter on complex, multi-hop queries that require synthesizing information from disparate sources. Current advanced RAG methods, employing iterative or adaptive strategies, lack a robust mechanism to systematically identify and fill evidence gaps, often propagating noise or failing to gather a comprehensive context. We introduce FAIR-RAG, a novel agentic framework that transforms the standard RAG pipeline into a dynamic, evidence-driven reasoning process. At its core is an Iterative Refinement Cycle governed by a module we term Structured Evidence Assessment (SEA). The SEA acts as an analytical gating mechanism: it deconstructs the initial query into a checklist of required findings and audits the aggregated evidence to identify confirmed facts and, critically, explicit informational gaps. These gaps provide a precise signal to an Adaptive Query Refinement agent, which generates new, targeted sub-queries to retrieve missing information. This cycle repeats until the evidence is verified as sufficient, ensuring a comprehensive context for a final, strictly faithful generation. We conducted experiments on challenging multi-hop QA benchmarks, including HotpotQA, 2WikiMultiHopQA, and MusiQue. In a unified experimental setup, FAIR-RAG significantly outperforms strong baselines. On HotpotQA, it achieves an F1-score of 0.453 -- an absolute improvement of 8.3 points over the strongest iterative baseline -- establishing a new state-of-the-art for this class of methods on these benchmarks. Our work demonstrates that a structured, evidence-driven refinement process with explicit gap analysis is crucial for unlocking reliable and accurate reasoning in advanced RAG systems for complex, knowledge-intensive tasks.

Figures (5)

• Figure 1: Schematic overview of the FAIR-RAG architecture. The process starts with initial query analysis and adaptive language model selection (e.g., small, large, or reasoning LLM). For complex queries, it proceeds to query decomposition, followed by an iterative refinement cycle involving hybrid retrieval and reranking, evidence filtering, and Structured Evidence Assessment (SEA). The loop iterates until evidence sufficiency is confirmed, culminating in faithful answer generation grounded in the aggregated evidence.
• Figure 2: Performance comparison of FAIR-RAG variants against baseline methods on four question-answering benchmarks. Each subplot displays the F1 score (bars, left axis) and the LLM-as-Judge Accuracy (ACCLLM, line, right axis). Our FAIR-RAG models are highlighted with a hatched pattern. The results consistently demonstrate the superiority of our framework, especially on complex multi-hop datasets (HotpotQA, 2WikiMultiHopQA, and MusiQue), where it significantly outperforms all baselines in F1 score while maintaining high semantic accuracy. All evaluations are conducted on 1000 samples from each benchmark's development set.
• Figure 3: A qualitative case study demonstrating FAIR-RAG's two-iteration process for a complex comparative query. In Iteration 1, the system decomposes the query and retrieves initial evidence identifying the museums but lacking the required architectural styles. The Structured Evidence Assessment (SEA) module correctly identifies this gap (Is Sufficient: No), triggering a second iteration. In Iteration 2, the system generates Refined Queries targeting the missing information, successfully retrieves the necessary evidence, and confirms sufficiency. The process culminates in a Final Faithful Generation that synthesizes evidence from both iterations into a comprehensive answer.
• Figure 4: Aggregate Distribution of Failure Sources. Analysis of 200 error samples reveals a primary split between Component-Level Failures (63.5%) and Architectural Failures (36.5%). While architectural logic offers direct avenues for refinement, the majority of errors stem from the inherent limitations of the foundational retrieval and generation models, identifying them as the principal bottleneck for the FAIR-RAG system.
• Figure 5: Task-Dependent Distribution of Failure Modes. The analysis highlights a strong correlation between task complexity and the primary failure bottleneck. For the factoid-centric TriviaQA, Retrieval Failures are dominant (47%). Conversely, for complex multi-hop reasoning datasets like MuSiQue, HotpotQA, and 2WikiMultihopQA, the burden shifts towards reasoning, with SEA Errors becoming a major failure category (28-32%). This trend validates the critical role of the strategic reasoning component (SEA) for successfully navigating multi-step queries.