Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model

Ling Team, Anqi Shen, Baihui Li, Bin Hu, Bin Jing, Cai Chen, Chao Huang, Chao Zhang, Chaokun Yang, Cheng Lin, Chengyao Wen, Congqi Li, Deng Zhao, Dingbo Yuan, Donghai You, Fagui Mao, Fanzhuang Meng, Feng Xu, Guojie Li, Guowei Wang, Hao Dai, Haonan Zheng, Hong Liu, Jia Guo, Jiaming Liu, Jian Liu, Jianhao Fu, Jiannan Shi, Jianwen Wang, Jianxin Lai, Jin Yang, Jun Mei, Jun Zhou, Junbo Zhao, Junping Zhao, Kuan Xu, Le Su, Lei Chen, Li Tang, Liang Jiang, Liangcheng Fu, Lianhao Xu, Linfeng Shi, Lisha Liao, Longfei Zheng, Meng Li, Mingchun Chen, Qi Zuo, Qiang Cheng, Qianggang Cao, Qitao Shi, Quanrui Guo, Senlin Zhu, Shaofei Wang, Shaomian Zheng, Shuaicheng Li, Shuwei Gu, Siba Chen, Tao Wu, Tao Zhang, Tianyu Zhang, Tianyu Zhou, Tiwei Bie, Tongkai Yang, Wang Hong, Wang Ren, Weihua Chen, Wenbo Yu, Wengang Zheng, Xiangchun Wang, Xiaodong Yan, Xiaopei Wan, Xin Zhao, Xinyu Kong, Xinyu Tang, Xudong Han, Xudong Wang, Xuemin Yang, Xueyu Hu, Yalin Zhang, Yan Sun, Yicheng Shan, Yilong Wang, Yingying Xu, Yongkang Liu, Yongzhen Guo, Yuanyuan Wang, Yuchen Yan, Yuefan Wang, Yuhong Guo, Zehuan Li, Zhankai Xu, Zhe Li, Zhenduo Zhang, Zhengke Gui, Zhenxuan Pan, Zhenyu Huang, Zhenzhong Lan, Zhiqiang Ding, Zhiqiang Zhang, Zhixun Li, Zhizhen Liu, Zihao Wang, Zujie Wen

TL;DR

Ring-1T advances the frontier of scalable thinking models by tackling reinforcement learning at the trillion-parameter scale. The authors introduce IcePop to curb training-inference discrepancy, C3PO++ to optimize long-rollout efficiency under a token budget, and ASystem as an integrated RL infrastructure to enable stable, large-scale training. Built on Ling 2.0 and trained from Ling-1T-base, Ring-1T achieves leading open-weight performance across mathematical reasoning, coding, and general reasoning tasks, including a silver-medal IMO-2025 result and AIME/HMMT top scores, while democratizing access to trillion-parameter reasoning capabilities. The work demonstrates that a combination of algorithmic safeguards, rollout-efficient scheduling, and a purpose-built RL stack can unlock robust, open, trillion-parameter reasoning models with practical impact for researchers and developers. Overall, Ring-1T establishes a new baseline for open-source large-scale thinking models and highlights a scalable path toward more capable AI systems.

Abstract

We present Ring-1T, the first open-source, state-of-the-art thinking model with a trillion-scale parameter. It features 1 trillion total parameters and activates approximately 50 billion per token. Training such models at a trillion-parameter scale introduces unprecedented challenges, including train-inference misalignment, inefficiencies in rollout processing, and bottlenecks in the RL system. To address these, we pioneer three interconnected innovations: (1) IcePop stabilizes RL training via token-level discrepancy masking and clipping, resolving instability from training-inference mismatches; (2) C3PO++ improves resource utilization for long rollouts under a token budget by dynamically partitioning them, thereby obtaining high time efficiency; and (3) ASystem, a high-performance RL framework designed to overcome the systemic bottlenecks that impede trillion-parameter model training. Ring-1T delivers breakthrough results across critical benchmarks: 93.4 on AIME-2025, 86.72 on HMMT-2025, 2088 on CodeForces, and 55.94 on ARC-AGI-1. Notably, it attains a silver medal-level result on the IMO-2025, underscoring its exceptional reasoning capabilities. By releasing the complete 1T parameter MoE model to the community, we provide the research community with direct access to cutting-edge reasoning capabilities. This contribution marks a significant milestone in democratizing large-scale reasoning intelligence and establishes a new baseline for open-source model performance.

Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model

TL;DR

Ring-1T advances the frontier of scalable thinking models by tackling reinforcement learning at the trillion-parameter scale. The authors introduce IcePop to curb training-inference discrepancy, C3PO++ to optimize long-rollout efficiency under a token budget, and ASystem as an integrated RL infrastructure to enable stable, large-scale training. Built on Ling 2.0 and trained from Ling-1T-base, Ring-1T achieves leading open-weight performance across mathematical reasoning, coding, and general reasoning tasks, including a silver-medal IMO-2025 result and AIME/HMMT top scores, while democratizing access to trillion-parameter reasoning capabilities. The work demonstrates that a combination of algorithmic safeguards, rollout-efficient scheduling, and a purpose-built RL stack can unlock robust, open, trillion-parameter reasoning models with practical impact for researchers and developers. Overall, Ring-1T establishes a new baseline for open-source large-scale thinking models and highlights a scalable path toward more capable AI systems.

Abstract

We present Ring-1T, the first open-source, state-of-the-art thinking model with a trillion-scale parameter. It features 1 trillion total parameters and activates approximately 50 billion per token. Training such models at a trillion-parameter scale introduces unprecedented challenges, including train-inference misalignment, inefficiencies in rollout processing, and bottlenecks in the RL system. To address these, we pioneer three interconnected innovations: (1) IcePop stabilizes RL training via token-level discrepancy masking and clipping, resolving instability from training-inference mismatches; (2) C3PO++ improves resource utilization for long rollouts under a token budget by dynamically partitioning them, thereby obtaining high time efficiency; and (3) ASystem, a high-performance RL framework designed to overcome the systemic bottlenecks that impede trillion-parameter model training. Ring-1T delivers breakthrough results across critical benchmarks: 93.4 on AIME-2025, 86.72 on HMMT-2025, 2088 on CodeForces, and 55.94 on ARC-AGI-1. Notably, it attains a silver medal-level result on the IMO-2025, underscoring its exceptional reasoning capabilities. By releasing the complete 1T parameter MoE model to the community, we provide the research community with direct access to cutting-edge reasoning capabilities. This contribution marks a significant milestone in democratizing large-scale reasoning intelligence and establishes a new baseline for open-source model performance.
Paper Structure (48 sections, 2 theorems, 27 equations, 13 figures, 1 table, 1 algorithm)

This paper contains 48 sections, 2 theorems, 27 equations, 13 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Approach
  3. Training Pipeline
  4. Long-CoT SFT
  5. Data Collection
  6. Training
  7. Large-Scale Reinforcement Learning
  8. RL Data
  9. IcePop: Discard All Noisy Gradient Updates
  10. C3PO++: Dynamically Partition Rollouts with Budget
  11. Training Recipe
  12. Experiments and Analysis
  13. IcePop
  14. Setup
  15. Preliminary results on Ring-mini-2.0.
  16. ...and 33 more sections

Key Result

Theorem 1

(Compounding Probability Discrepancy) Let $\pi_{\mathrm{infer}}(\cdot;\theta)$ and $\pi_{\mathrm{train}}(\cdot;\theta)$ be the policy model loaded by inference and training engines, and $\delta_t \;=\; D_{\mathrm{KL}}\!(\pi_{\mathrm{infer}}(\cdot;\theta_t)\,\|\,\pi_{\mathrm{train}}(\cdot;\theta_t))$

Figures (13)

  • Figure 1: Performance comparison of Ring-1T and existing open-weights and close-weights$^{\dagger}$ models across benchmarks.
  • Figure 2: The training pipeline of Ring-1T.
  • Figure 3: We integrate C3PO++ and IcePop into Ring-1T, which enhances both training efficiency and effectiveness of RL.
  • Figure 4: C3PO++ improves reinforcement learning efficiency for large thinking models by maintaining a rollout buffer across policy model versions. Once the rollout in an iteration reaches the token budget, optimization is performed; unfinished rollouts are stored in the buffer and resumed by the updated policy in the next iteration.
  • Figure 5: The performance comparison on AIME25 (Avg@64). We evaluate all models using the same setting.
  • ...and 8 more figures

Theorems & Definitions (3)

  • Theorem 1
  • Theorem 2: Compounding probability discrepancy
  • proof