Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Context Verification: Hierarchical Detection of Benchmark Contamination through Session-Isolated Analysis

Tae-Eun Song

Abstract

LLM coding benchmarks face a credibility crisis: widespread solution leakage and test quality issues undermine SWE-bench Verified, while existing detection methods--paraphrase consistency, n-gram overlap, perplexity analysis--never directly observe whether a model reasons or recalls. Meanwhile, simply repeating verification degrades accuracy: multi-turn review generates false positives faster than it discovers true errors, suggesting that structural approaches are needed. We introduce Cross-Context Verification (CCV), a black-box method that solves the same benchmark problem in N independent sessions and measures solution diversity, combined with the Hierarchical Cross-Context Architecture (HCCA), a multi-agent analysis framework that prevents confirmation bias through intentional information restriction across specialized analytical roles. On 9 SWE-bench Verified problems (45 trials, Claude Opus 4.6, temperature 0), CCV achieves perfect separation between contaminated and genuine reasoning (Mann-Whitney U=0, p approx 0.012, r = 1.0). Key findings: (1) contamination is binary--models either recall perfectly or not at all; (2) reasoning absence is a perfect discriminator; (3) 33% of prior contamination labels are false positives; (4) HCCA's independent analysis structure discovers contamination-flaw composite cases that single-analyst approaches miss. A pilot experiment extending HCCA to multi-stage verification (Worker to Verifier to Director) yields a negative result--100% sycophantic confirmation--providing further evidence that information restriction, not structural complexity, is the key mechanism. We release all code and data.

Cross-Context Verification: Hierarchical Detection of Benchmark Contamination through Session-Isolated Analysis

Abstract

LLM coding benchmarks face a credibility crisis: widespread solution leakage and test quality issues undermine SWE-bench Verified, while existing detection methods--paraphrase consistency, n-gram overlap, perplexity analysis--never directly observe whether a model reasons or recalls. Meanwhile, simply repeating verification degrades accuracy: multi-turn review generates false positives faster than it discovers true errors, suggesting that structural approaches are needed. We introduce Cross-Context Verification (CCV), a black-box method that solves the same benchmark problem in N independent sessions and measures solution diversity, combined with the Hierarchical Cross-Context Architecture (HCCA), a multi-agent analysis framework that prevents confirmation bias through intentional information restriction across specialized analytical roles. On 9 SWE-bench Verified problems (45 trials, Claude Opus 4.6, temperature 0), CCV achieves perfect separation between contaminated and genuine reasoning (Mann-Whitney U=0, p approx 0.012, r = 1.0). Key findings: (1) contamination is binary--models either recall perfectly or not at all; (2) reasoning absence is a perfect discriminator; (3) 33% of prior contamination labels are false positives; (4) HCCA's independent analysis structure discovers contamination-flaw composite cases that single-analyst approaches miss. A pilot experiment extending HCCA to multi-stage verification (Worker to Verifier to Director) yields a negative result--100% sycophantic confirmation--providing further evidence that information restriction, not structural complexity, is the key mechanism. We release all code and data.
Paper Structure (73 sections, 3 equations, 3 figures, 4 tables)

This paper contains 73 sections, 3 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Key insight.
  3. From repetition to structure.
  4. Contributions.
  5. Related Work
  6. Benchmark contamination detection.
  7. SWE-bench test quality.
  8. Multi-agent verification.
  9. Multi-turn verification.
  10. Method: Cross-Context Verification
  11. Session Isolation
  12. Session isolation.
  13. Core assumption.
  14. Diversity Metrics
  15. Code diversity.
  16. ...and 58 more sections

Figures (3)

  • Figure 1: CCV pipeline. A benchmark problem is solved $N$ times in session-isolated trials. Solution diversity and gold proximity are analyzed to produce contamination and test flaw scores.
  • Figure 2: Contamination score distribution. Red bars: contaminated behavior ($\text{CS} \geq 0.6$); blue bars: genuine reasoning ($\text{CS} < 0.6$). The dashed line at 0.6 achieves perfect separation with no overlap.
  • Figure 3: Code diversity vs. gold BLEU for 9 problems. Three clusters emerge: contaminated (top-left, zero diversity, high gold match), contamination-flaw composite (bottom-left, zero diversity, low gold match), and genuine reasoning (right, high diversity).