Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Literal Extraction, Logical Inference, and Hallucination Risks in Long-Context LLMs

TL;DR

This work probes the reliability of long-context LLMs by extending the needle-in-a-haystack benchmark to include realistic fact distributions and explicit anti-hallucination prompts. It separates literal extraction, logical inference, and hallucination risk across four production-scale models, revealing that longer context can hurt performance when evidence is dispersed and that safety prompts can induce over-refusal, degrading extraction and reasoning. A key contribution is the identification of distributional robustness and the so-called distributional collapse, where central clustering of evidence cripples retrieval and inference in some models. The findings suggest that architectural strategies promoting semantic continuity and robust attention are crucial, and they highlight the continued relevance of grounding approaches (e.g., RAG) for enterprise-scale long-context applications.

Abstract

Large language models (LLMs) increasingly support very long input contexts. Yet it remains unclear how reliably they extract and infer information at scale. Performance varies with context length and strongly interacts with how information is distributed in real-world corpora. Motivated by these observations, we study how fact placement, corpus-level fact distributions, and Don't Make It Up prompts influence model behavior. We introduce an extended needle-in-a-haystack benchmark across four production-scale models: Gemini-2.5-flash, ChatGPT-5-mini, Claude-4.5-haiku, and Deepseek-v3.2-chat. Unlike prior work, we separately evaluate literal extraction, logical inference, and hallucination risk. Our study considers both positional effects and realistic distributions of evidence across long contexts, as well as prompts that explicitly discourage fabrication. We find that longer contexts alone do not guarantee better performance and can be detrimental when relevant evidence is diluted or widely dispersed. Performance varies substantially across models: some show severe degradation under realistic conditions, while others remain more robust at longer context lengths. Anti-hallucination (AH) instructions can make some models overly conservative, sharply reducing accuracy in literal extraction and logical inference. While we do not directly compare retrieval-augmented generation (RAG) and cache-augmented generation (CAG), our results suggest many failures stem from ineffective context utilization. Models often struggle to identify and prioritize relevant information even when it is present. These findings have direct practical implications, as enterprise workflows increasingly involve pasting large volumes of unfiltered documents into LLM prompts. Effective context length and model-specific robustness to long contexts are therefore critical for reliable LLM deployment in research and business.

Figures (8)

• Figure 1: Overview of the Extended Needle in a Haystack Evaluation Framework. (A) Uniform sweeps across information depth and context length establish baseline performance maps. (B) Probabilistic fact placement strategies (e.g., Normal or Exponential) simulate the informational dispersion characteristic of real-world documents. The framework evaluates three capabilities: Literal Extraction, Logical Inference, and Faithfulness, under both Standard and Anti-Hallucination prompting conditions.
• Figure 2: Average performance scaling across context lengths (log scale). For each context length, the reported performance is averaged across all tested fact placement depths. Solid lines represent Literal Extraction, dashed lines represent Logical Inference, and dotted lines represent Faithfulness. While Gemini-2.5-flash and Deepseek-v3.2-chat remain stable, other models show significant degradation as token counts increase.
• Figure 3: Performance sensitivity to information depth. The x-axis represents the relative position of the fact within the context (0% = start, 100% = end). Curves are smoothed averages across all tested context lengths, highlighting positional biases such as the "lost-in-the-middle" phenomenon.
• Figure 4: Literal Extraction accuracy heatmaps comparing (a) Standard Prompts and (b) Anti-Hallucination Prompts. The x-axis represents context length (normalized), and the y-axis represents depth. Darker green indicates higher accuracy, while red indicates failure. Note the emergence of significant failure regions in ChatGPT-5-mini under the Anti-Hallucination condition.
• Figure 5: Performance delta ($\Delta$) heatmaps showing the shift in accuracy when applying Anti-Hallucination prompts. Red indicates performance degradation (over-refusal), while blue indicates improvement. ChatGPT-5-mini (Column 2) exhibits severe degradation in Literal Extraction and Logical Inference, contrasting with the Faithfulness gains in Panel (c). Saturation is capped at ±30% to highlight subtle performance variances.