Revealing the Learning Dynamics of Long-Context Continual Pre-training

Abstract

Existing studies on Long-Context Continual Pre-training (LCCP) mainly focus on small-scale models and limited data regimes (tens of billions of tokens). We argue that directly migrating these small-scale settings to industrial-grade models risks insufficient adaptation and premature training termination. Furthermore, current evaluation methods rely heavily on downstream benchmarks (e.g., Needle-in-a-Haystack), which often fail to reflect the intrinsic convergence state and can lead to "deceptive saturation". In this paper, we present the first systematic investigation of LCCP learning dynamics using the industrial-grade Hunyuan-A13B (80B total parameters), tracking its evolution across a 200B-token training trajectory. Specifically, we propose a hierarchical framework to analyze LCCP dynamics across behavioral (supervised fine-tuning probing), probabilistic (perplexity), and mechanistic (attention patterns) levels. Our findings reveal: (1) Necessity of Massive Data Scaling: Training regimes of dozens of billions of tokens are insufficient for industrial-grade LLMs' LCCP (e.g., Hunyuan-A13B reaches saturation after training over 150B tokens). (2) Deceptive Saturation vs. Intrinsic Saturation: Traditional NIAH scores report "fake saturation" early, while our PPL-based analysis reveals continuous intrinsic improvements and correlates more strongly with downstream performance. (3) Mechanistic Monitoring for Training Stability: Retrieval heads act as efficient, low-resource training monitors, as their evolving attention scores reliably track LCCP progress and exhibit high correlation with SFT results. This work provides a comprehensive monitoring framework, evaluation system, and mechanistic interpretation for the LCCP of industrial-grade LLM.

Figures (9)

• Figure 1: The performance on the RULER, MRCR, and LongBio benchmarks of different LCCP checkpoints of the Hunyuan-A13B model after lightweight SFT.
• Figure 2: The PPL distribution for NIAH tasks across different context lengths and depths for Hunyuan-A13B base model at different stages of LCCP. Each cell represents the average PPL of the answer tokens.
• Figure 3: Average NIAH score and PPL across Hunyuan-A13B base model at different stages of LCCP.
• Figure 4: The retrieval scores of all attention heads across all layers of the Hunyuan-A13B base model at different stages of LCCP.
• Figure 5: The number of retrieval heads and the average retrieval score of the initial retrieval heads of the Hunyuan-A13B base model at different stages of LCCP.