Sufficient Context: A New Lens on Retrieval Augmented Generation Systems

TL;DR

The paper introduces a formal notion of sufficient context for retrieval-augmented generation, coupled with an autorater to label whether a given query-context pair can support an answer. By stratifying model outputs by context sufficiency across multiple datasets and models, it reveals that larger models still commonly hallucinate even when sufficient context is present and that some correct answers arise even with insufficient context. It also demonstrates a selective-generation approach that combines sufficient-context signals with model confidence to reduce hallucinations and improve accuracy by 2–10% on several models and datasets. The work suggests that improving RAG performance requires both better retrieval and smarter, context-aware generation strategies, and it provides practical prompts and evaluation tools (autorater) to analyze context-dependent behavior. Overall, the study advances understanding of how context quality affects RAG systems and offers actionable techniques to mitigate hallucinations in real-world deployments.

Abstract

Augmenting LLMs with context leads to improved performance across many applications. Despite much research on Retrieval Augmented Generation (RAG) systems, an open question is whether errors arise because LLMs fail to utilize the context from retrieval or the context itself is insufficient to answer the query. To shed light on this, we develop a new notion of sufficient context, along with a method to classify instances that have enough information to answer the query. We then use sufficient context to analyze several models and datasets. By stratifying errors based on context sufficiency, we find that larger models with higher baseline performance (Gemini 1.5 Pro, GPT 4o, Claude 3.5) excel at answering queries when the context is sufficient, but often output incorrect answers instead of abstaining when the context is not. On the other hand, smaller models with lower baseline performance (Mistral 3, Gemma 2) hallucinate or abstain often, even with sufficient context. We further categorize cases when the context is useful, and improves accuracy, even though it does not fully answer the query and the model errs without the context. Building on our findings, we explore ways to reduce hallucinations in RAG systems, including a new selective generation method that leverages sufficient context information for guided abstention. Our method improves the fraction of correct answers among times where the model responds by 2--10\% for Gemini, GPT, and Gemma. Key findings and the prompts used in our autorater analysis are available on our github.

Figures (6)

• Figure 1: New insights into RAG systems by looking at whether instances have sufficient context. On the left, we show examples of sufficient context; on the right, a breakdown of model responses on the Musique dataset. Adding RAG improves the percentage of correct answers. Unfortunately, with RAG, models hallucinate more than abstain, and the insufficiency of the context does not account for this major issue. Also, standard datasets have many instances with insufficient context (here, 55.4%). We include results for other datasets (FreshQA, HotPotQA) in Appendix \ref{['appendix:additional']}, showing similar trends.
• Figure 2: We compare the % of instances that our autorater labels as sufficient across datasets, either with the first 10k, 6k, or 2k tokens of the provided sources. FreshQA has hand-curated URLs that support the answers and exhibits high sufficient context. HotPotQA and Musique have lower sufficient context (and even lower with 2000 tokens). We use 6000 token contexts in the remainder.
• Figure 3: Model Performance on Datasets Stratified by Sufficient Context. Given sufficient context, models have a higher correct percentage on these challenging datasets. Performance drops, but the models are still able to answer a large portion of questions correct without sufficient context. One prevailing issue is that all models hallucinate rather than abstain in many cases with insufficient context. The smallest model Gemma 27B struggles to avoid hallucinations given insufficient context.
• Figure 4: Selective Generation: Coverage vs. Selective Accuracy. For selective generation, we use a linear combination of sufficient context and self-rated confidence (purple) or confidence alone (gray). The x-axis shows coverage (% of questions answered); the y-axis shows accuracy at each coverage (# correct / # answered). The combined approach matches or outperforms the baseline confidence-only method, especially on HotpotQA, where our method improves accuracy for most coverages. For Gemma 27B on Musique, the methods are identical (coeff. for stuff. context is 0).
• Figure 5: Correct, hallucination, and abstention fractions across models for dataset FreshQA, stratified by sufficient context. FreshQA includes hand-curated source URLs, which explains the larger percentage of sufficient context (77.4%). FreshQA also specifically explores questions with answers that change based on the question's timestamp, which may explain the frequent abstentions without RAG (100% for Gemini 1.5 Pro).