Zero-RAG: Towards Retrieval-Augmented Generation with Zero Redundant Knowledge

TL;DR

Zero-RAG targets redundancy between a large external corpus and expanding LLM internal knowledge in Retrieval-Augmented Generation (RAG). It introduces Mastery-Score to prune redundant passages, and adds a Query Router and Noise-Tolerant Tuning to maximize reliance on internal knowledge after pruning. Across four QA benchmarks, pruning about 30% of Wikipedia reduces indexing and retrieval costs by roughly 22% without sacrificing performance, with ablations confirming the importance of each component. This approach yields more scalable RAG by trimming redundant data while preserving answer quality, enabling faster and more robust retrieval-augmented inference.

Abstract

Retrieval-Augmented Generation has shown remarkable results to address Large Language Models' hallucinations, which usually uses a large external corpus to supplement knowledge to LLMs. However, with the development of LLMs, the internal knowledge of LLMs has expanded significantly, thus causing significant knowledge redundancy between the external corpus and LLMs. On the one hand, the indexing cost of dense retrieval is highly related to the corpus size and thus significant redundant knowledge intensifies the dense retrieval's workload. On the other hand, the redundant knowledge in the external corpus is not helpful to LLMs and our exploratory analysis shows that it instead hurts the RAG performance on those questions which the LLM can answer by itself. To address these issues, we propose Zero-RAG to tackle these challenges. Specifically, we first propose the Mastery-Score metric to identify redundant knowledge in the RAG corpus to prune it. After pruning, answers to "mastered" questions rely primarily on internal knowledge of the LLM. To better harness the internal capacity, we propose Query Router and Noise-Tolerant Tuning to avoid the irrelevant documents' distraction and thus further improve the LLM's utilization of internal knowledge with pruned corpus. Experimental results show that Zero-RAG prunes the Wikipedia corpus by 30\% and accelerates the retrieval stage by 22\%, without compromising RAG's performance.

Figures (4)

• Figure 1: Overview of knowledge redundancy: (a) shows the overlap between LLM and corpus knowledge; (b) indicates that a large part of LLM knowledge is derived from Wikipedia; (c) reveals that redundant knowledge degrades performance on originally correct responses.
• Figure 2: Relationship between model‐internal knowledge and corpus knowledge. The left circle denotes the knowledge already encoded in the language model ($\mathcal{K}_{\mathcal{M}}$), while the right circle denotes the out-of-model corpus $\mathcal{D}$. Their intersection, $\mathcal{D}_{\text{redundant}}$, contains facts duplicated in both sources and is therefore a prime target for pruning during Zero-RAG
• Figure 3: Mastery-Score Construction Pipeline: For a given sentence, the mastery score is computed in four steps: (1) sampling candidate sentences from Wikipedia; (2) generating $n$ QA pairs for each sentence; (3) evaluating LLM performance on these QA pairs; and (4) calculating the final score based on the LLM's responses.
• Figure 4: Overall architecture of Zero-RAG. The pipeline comprises four key stages: (1) Corpus Pruning via Mastery-Score, which filters out already-mastered documents; (2) Query Router, which dynamically decides whether to retrieve or not; (3) Noise-Tolerant Tuning, ensuring robustness against partially relevant or irrelevant documents; and (4) the final Inference stage that integrates all previous steps.