Simulated Ignorance Fails: A Systematic Study of LLM Behaviors on Forecasting Problems Before Model Knowledge Cutoff

TL;DR

The paper investigates whether Simulated Ignorance (SI) prompts can make pre-cutoff forecasting questions behave like True Ignorance (TI) in LLMs. Across 477 competition-like questions and 9 models, three increasingly strong interventions—cutoff instructions, chain-of-thought prompting, and reasoning-optimized models—reduce explicit leakage but fail to close the gap, leaving SI outperform TI by a substantial margin. The authors quantify the remaining gap as a mean SI–TI difference of about $52\%$ of the original, with $48\%$ of the gap closed, and provide robust cross-model, domain, and qualitative evidence that the leakage is structural, not merely surface-level. They conclude that SI-based retrospective evaluation is methodologically flawed for forecasting and urge relying on genuinely post-cutoff (TI) events or continuously refreshed benchmarks; they also discuss implications for reasoning faithfulness and evaluation practices in LLMs. The study thus highlights the practical limits of prompt-based unlearning and calls for moving beyond SI to ensure credible forecasting benchmarks, with explicit recommendations for dataset design and evaluation metrics. $B(p,y)=(p-y)^2$ is used to calibrate forecasts, and representations of leakage are supported by analyses of $K$-cutoffs, cross-model comparisons, and human baselines, all of which reinforce that SI cannot replicate TI in forecasting tasks.

Abstract

Evaluating LLM forecasting capabilities is constrained by a fundamental tension: prospective evaluation offers methodological rigor but prohibitive latency, while retrospective forecasting (RF) -- evaluating on already-resolved events -- faces rapidly shrinking clean evaluation data as SOTA models possess increasingly recent knowledge cutoffs. Simulated Ignorance (SI), prompting models to suppress pre-cutoff knowledge, has emerged as a potential solution. We provide the first systematic test of whether SI can approximate True Ignorance (TI). Across 477 competition-level questions and 9 models, we find that SI fails systematically: (1) cutoff instructions leave a 52% performance gap between SI and TI; (2) chain-of-thought reasoning fails to suppress prior knowledge, even when reasoning traces contain no explicit post-cutoff references; (3) reasoning-optimized models exhibit worse SI fidelity despite superior reasoning trace quality. These findings demonstrate that prompts cannot reliably "rewind" model knowledge. We conclude that RF on pre-cutoff events is methodologically flawed; we recommend against using SI-based retrospective setups to benchmark forecasting capabilities.

Figures (5)

• Figure 2: Layer 1: Cutoff instructions reduce but do not eliminate the SI--TI gap.(A) Brier scores across 9 models under three conditions: Pre-cutoff baseline (Pre, G1), SI (Pre, G1$'$), and TI (Post, G1$'$). (B) Gap decomposition by domain: portion closed by cutoff (green) vs. residual SI--TI gap (red). Lower Brier = better.
• Figure 3: Layer 2: CoT prompting effects. (A) Without cutoff instruction: CoT worsens Pre-cutoff but improves Post-cutoff, narrowing the baseline gap. (B) With cutoff instruction: G1$'$$\to$ G2$'$ yields small changes on both SI and TI.
• Figure 4: Layer 2: Trace compliance. (A) Cutoff penalty on Pre-cutoff is larger for direct prompting than CoT. (B) Trace audit metrics improve from G2 to G2$'$.
• Figure 5: Layer 3: Reasoning-optimized models show cleaner traces but larger SI--TI gaps. (A) Brier scores across prompting conditions. (B) SI--TI gap comparison: reasoning-optimized vs. non-reasoning models.
• Figure 6: Discontinuity at model-specific cutoffs (G2$'$). Each model shows a sharp Brier increase at its own cutoff (vertical line), not at a common calendar point. (A) GPT-5.1 cutoff: September 2024. (B) DeepSeek-V3.2 cutoff: January 2025.