Intelligence Degradation in Long-Context LLMs: Critical Threshold Determination via Natural Length Distribution Analysis
Weiwei Wang, Jiyong Min, Weijie Zou
TL;DR
This work identifies a cliff-like intelligence degradation in long-context LLMs, formalizing intelligence as $I(L)=\text{F1}(L)$ and showing a stable regime up to roughly 40% of the 128K maximum context, followed by a sharp collapse near 43–50% where F1 falls by about $45.5\%$. The authors propose a natural-length distribution analysis, a five-method cross-validation framework for precise threshold detection, and a unified framework of shallow long-context adaptation that links training biases, RoPE extrapolation limits, and attention dispersion. Empirical results on Qwen2.5-7B using a mixed SQuAD/NarrativeQA dataset demonstrate a robust threshold at $L_c\approx0.432$ of max context, with three distinct performance regions and strong statistical support ($p<0.001$, $r=-0.68$ in the transition). The work provides practical deployment guidance (remain below ~40% of max length for stability) and a rigorous methodological blueprint for identifying critical length thresholds across models, tasks, and architectures, advancing open-source understanding of long-context capabilities.
Abstract
Large Language Models (LLMs) exhibit catastrophic performance degradation when processing contexts approaching certain critical thresholds, even when information remains relevant. This intelligence degradation-defined as over 30% drop in task performance-severely limits long-context applications. This degradation shows a common pattern: models maintain strong performance up to a critical threshold, then collapse catastrophically. We term this shallow long-context adaptation-models adapt for short to medium contexts but fail beyond critical thresholds. This paper presents three contributions: (1) Natural Length Distribution Analysis: We use each sample's natural token length without truncation or padding, providing stronger causal evidence that degradation results from context length itself. (2) Critical Threshold Determination: Through experiments on a mixed dataset (1,000 samples covering 5%-95% of context length), we identify the critical threshold for Qwen2.5-7B at 40-50% of maximum context length, where F1 scores drop from 0.55-0.56 to 0.3 (45.5% degradation), using five-method cross-validation. (3) Unified Framework: We consolidate shallow adaptation, explaining degradation patterns and providing a foundation for mitigation strategies. This work provides the first systematic characterization of intelligence degradation in open-source Qwen models, offering practical guidance for deploying LLMs in long-context scenarios.