HE-SNR: Uncovering Latent Logic via Entropy for Guiding Mid-Training on SWE-BENCH

TL;DR

The paper tackles the challenge of guiding mid-training for SWE-focused LLMs by identifying the inadequacy of perplexity under the Long-Context Tax and proposing an entropy-centric framework. Grounded in the Entropy Compression Hypothesis, it introduces HE-SNR, a metric that concentrates on high-entropy decision points within a carefully filtered action space, and demonstrates a strong, linear relationship between HE-SNR (and HE-PPL) and downstream SWE-bench performance across large MoE models and extended context windows. It reveals that SFT can incur an Alignment Tax, degrading high-entropy predictive signals even as global PPL improves, and provides data-efficient evaluation protocols using 500 trajectories. The work offers practical tools and theoretical insight for steering mid-training toward latent reasoning capabilities in complex software engineering tasks, with potential implications for efficiency and ecological impact. Overall, HE-SNR emerges as a robust, interpretable proxy for latent SWE potential, enabling more effective mid-training optimization and highlighting the nuanced trade-offs of alignment processes.

Abstract

SWE-bench has emerged as the premier benchmark for evaluating Large Language Models on complex software engineering tasks. While these capabilities are fundamentally acquired during the mid-training phase and subsequently elicited during Supervised Fine-Tuning (SFT), there remains a critical deficit in metrics capable of guiding mid-training effectively. Standard metrics such as Perplexity (PPL) are compromised by the "Long-Context Tax" and exhibit weak correlation with downstream SWE performance. In this paper, we bridge this gap by first introducing a rigorous data filtering strategy. Crucially, we propose the Entropy Compression Hypothesis, redefining intelligence not by scalar Top-1 compression, but by the capacity to structure uncertainty into Entropy-Compressed States of low orders ("reasonable hesitation"). Grounded in this fine-grained entropy analysis, we formulate a novel metric, HE-SNR (High-Entropy Signal-to-Noise Ratio). Validated on industrial-scale Mixture-of-Experts (MoE) models across varying context windows (32K/128K), our approach demonstrates superior robustness and predictive power. This work provides both the theoretical foundation and practical tools for optimizing the latent potential of LLMs in complex engineering domains.

Figures (11)

• Figure 1: Correlation between PPL and Top-$k$ accuracy, evaluated on the LLM-generated components (Thought and Action) of the curated SWE-bench test dataset. Annotations (e.g., 500, 2k) indicate the training step count for specific checkpoints.
• Figure 2: Evolution of metrics (on filtered Action tokens) vs. SWE-bench Pass@1 during 128K extension. "Step 0" marks the pre-RoPE adjustment 32K baseline. (a) MoE-S. (b) MoE-L. Note the inverse correlation in (b): SWE performance improves despite PPL/Top-$1$ degradation caused by the Long-Context Tax.
• Figure 3: Top-$10$ entropy distributions for MoE-S: Base models (Left) vs. Post-SFT models (Right) across 32K/128K checkpoints. Red dashed lines mark $\ln 2$ to $\ln 5$. The brown vertical line in Non-Top-2 plots indicates the global peak (mode).
• Figure 4: Top-$10$ entropy distributions for MoE-L: Base models (Left) vs. Post-SFT models (Right) at selected 32K and 128K checkpoints.
• Figure 5: Top-$10$ entropy distributions for the MoE-L model computed on Observation tokens. The distribution exhibits distinct peaks near $\ln 2$ and $\ln 10$, contrasting with the structure observed in Action tokens.

Theorems & Definitions (3)

• Lemma 4.1: Maximum Entropy of Top-$k$ Distribution
• Definition 4.2: Entropy-Compressed State
• proof