QuCo-RAG: Quantifying Uncertainty from the Pre-training Corpus for Dynamic Retrieval-Augmented Generation

TL;DR

QuCo-RAG replaces unreliable model-internal uncertainty signals with corpus-grounded uncertainty derived from pre-training data to dynamically trigger retrieval. It uses a two-stage approach: pre-generation entity-frequency checks and runtime co-occurrence verification, powered by Infini-gram over a trillion-token corpus. The method achieves state-of-the-art EM gains on multi-hop QA across model scales and transfers effectively to models with undisclosed training data, including biomedical domain generalization. This corpus-grounded verification paradigm offers a practical, model-agnostic path to mitigate hallucinations in dynamic RAG systems and informs data-centric AI strategies.

Abstract

Dynamic Retrieval-Augmented Generation adaptively determines when to retrieve during generation to mitigate hallucinations in large language models (LLMs). However, existing methods rely on model-internal signals (e.g., logits, entropy), which are fundamentally unreliable because LLMs are typically ill-calibrated and often exhibit high confidence in erroneous outputs. We propose QuCo-RAG, which shifts from subjective confidence to objective statistics computed from pre-training data. Our method quantifies uncertainty through two stages: (1) before generation, we identify low-frequency entities indicating long-tail knowledge gaps; (2) during generation, we verify entity co-occurrence in the pre-training corpus, where zero co-occurrence often signals hallucination risk. Both stages leverage Infini-gram for millisecond-latency queries over 4 trillion tokens, triggering retrieval when uncertainty is high. Experiments on multi-hop QA benchmarks show QuCo-RAG achieves EM gains of 5--12 points over state-of-the-art baselines with OLMo-2 models, and transfers effectively to models with undisclosed pre-training data (Llama, Qwen, GPT), improving EM by up to 14 points. Domain generalization on biomedical QA further validates the robustness of our paradigm. These results establish corpus-grounded verification as a principled, practically model-agnostic paradigm for dynamic RAG. Our code is publicly available at https://github.com/ZhishanQ/QuCo-RAG.

Figures (7)

• Figure 1: Comparison of retrieval triggering mechanisms. (a) DRAGIN relies on model-internal signals, incorrectly assigning high uncertainty to "Il" (a token from the question) while showing low uncertainty on the hallucinated director name. (b) QuCo-RAG correctly detects the hallucination through zero entity co-occurrence in the pre-training corpus.
• Figure 2: Overview of QuCo-RAG Framework.
• Figure 3: Efficiency-performance trade-off analysis on HotpotQA with OLMo-2-13B-Instruct. (a) EM score versus Token consumption. (b) EM score versus LLM calls. (c) Performance versus Retrieval frequency. QuCo-RAG achieves the highest EM with moderate token usage and LLM calls.
• Figure 4: Average runtime breakdown per question for QuCo-RAG components across OLMo-2 model sizes on 2WikiMultihopQA.
• Figure 5: Performance stratified by entity frequency bins on 2WikiMultihopQA (OLMo-2-7B).