Beyond the Needle's Illusion: Decoupled Evaluation of Evidence Access and Use under Semantic Interference at 326M-Token Scale

TL;DR

This work addresses the challenge of evaluating evidence access in long-context LLM usage at scale, where relying on benign Needle-in-a-Haystack benchmarks yields misleading robustness. It introduces EverMemBench-S (EMB-S), an adversarial NIAH-style benchmark built on a 326M-token MemoryBank with dense near-miss distractors and gold evidence spanning one or more documents, paired with a decoupled protocol that reports document-ID localization separately from end-to-end QA quality. Through a multi-scale reference corpus ladder, EMB-S demonstrates that semantic interference, not mere context length, becomes the dominant bottleneck for evidence access, especially for multi-source retrieval, and that robust evaluation benefits from a shared evidence interface across native long-context prompting and retrieval pipelines. The findings underscore the need for stronger evidence discrimination and diagnostic tools to assess and improve long-context memory systems in realistic, high-interference environments.

Abstract

Long-context LLM agents must access the right evidence from large environments and use it faithfully. However, the popular Needle-in-a-Haystack (NIAH) evaluation mostly measures benign span localization. The needle is near-unique, and the haystack is largely irrelevant. We introduce EverMemBench-S (EMB-S), an adversarial NIAH-style benchmark built on a 326M-token MemoryBank. While the full MemoryBank spans 326M tokens for retrieval-based (RAG) evaluation, we evaluate native long-context models only at scales that fit within each model's context window (up to 1M tokens in this work) to ensure a fair comparison. EMB-S pairs queries with collision-tested near-miss hard negatives and gold evidence sets spanning one or more documents, validated via human screening and LLM verification. We also propose a decoupled diagnostic protocol that reports evidence access (document-ID localization) separately from end-to-end QA quality under full-context prompting. This enables consistent diagnosis for both native long-context prompting and retrieval pipelines. Across a reference-corpus ladder from domain-isolated 64K contexts to a globally shared 326M-token environment, we observe a clear reality gap. Systems that saturate benign NIAH degrade sharply in evidence access under semantic interference. These results indicate that semantic discrimination, not context length alone, is the dominant bottleneck for long-context memory at scale.

Figures (6)

• Figure 1: Benign NIAH vs. EMB-S. NIAH finds a near-unique needle in mostly irrelevant content; EMB-S must distinguish near-miss negatives and integrate evidence that may span multiple documents.
• Figure 2: EMB-S overview. Component A builds 483 domain-labeled queries from a 326M-token MemoryBank (standardization, multi-hop synthesis, and collision testing). Component B defines a reference-corpus ladder from 64K to 326M tokens with increasing inter-domain mixing and distractor injection.
• Figure 3: Distribution of task types in EverMemBench-S and the distribution of context lengths for each task type.
• Figure 4: Cross-source mixing (512K vs. 326M). Distribution of the number of distinct datasets in the top-10 retrieved documents per query.
• Figure 5: Retrieval heatmap across the reference corpus ladder (64K--326M): evidence access degrades as the searchable pool expands and semantic interference increases.