EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control
Kai Yang, Xin Xu, Yangkun Chen, Weijie Liu, Jiafei Lyu, Zichuan Lin, Deheng Ye, Saiyong Yang
TL;DR
The paper tackles unstable entropy dynamics in long-term RL training of LLMs by proposing EntroPIC, a method that uses Proportional-Integral control to adaptively weight positive versus negative samples to keep policy entropy near a target $\mathcal{H}_{tar}$. The authors establish theoretical convergence for on-policy training with P/PI control and for off-policy training with PI control, and validate the approach through large-scale experiments showing stable entropy and improved performance across multiple math-oriented benchmarks. Key innovations include deriving how positive and negative samples affect entropy under binary rewards, and simplifying loss via high-probability token weighting without sacrificing convergence. Practically, EntroPIC enables stable, scalable RL for LLMs in industrial settings, offering plug-and-play applicability and robustness to temperature variations and late-stage entropy decline.
Abstract
Long-term training of large language models (LLMs) requires maintaining stable exploration to prevent the model from collapsing into sub-optimal behaviors. Entropy is crucial in this context, as it controls exploration and helps avoid premature convergence to sub-optimal solutions. However, existing reinforcement learning methods struggle to maintain an appropriate level of entropy, as the training process involves a mix of positive and negative samples, each affecting entropy in different ways across steps. To address this, we propose Entropy stablilization via Proportional-Integral Control (EntroPIC), a novel method that adaptively adjusts the influence of positive and negative samples by dynamically tuning their loss coefficients. This approach stabilizes entropy throughout training, ensuring efficient exploration and steady progress. We provide a comprehensive theoretical analysis for both on-policy and off-policy learning settings, demonstrating that EntroPIC is effective at controlling entropy in large-scale LLM training. Experimental results show that our method successfully maintains desired entropy levels, enabling stable and optimal RL training for LLMs.