Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation

Ling Team, Ang Li, Ben Liu, Binbin Hu, Bing Li, Bingwei Zeng, Borui Ye, Caizhi Tang, Changxin Tian, Chao Huang, Chao Zhang, Chen Qian, Chenchen Ju, Chenchen Li, Chengfu Tang, Chilin Fu, Chunshao Ren, Chunwei Wu, Cong Zhang, Cunyin Peng, Dafeng Xu, Daixin Wang, Dalong Zhang, Dingnan Jin, Dingyuan Zhu, Dongke Hu, Fangzheng Zhao, Feifan Wu, Feng Zhu, Gangshan Wang, Haitao Zhang, Hailin Zhao, Hanxiao Zhang, Hanzi Wang, Hao Qian, Haoyi Yu, Heng Zhang, Hongliang Zhang, Hongzhi Luan, Huirong Dong, Huizhong Li, Jia Li, Jia Liu, Jialong Zhu, Jian Sha, Jianping Wei, Jiaolong Yang, Jieyue Ma, Jiewei Wu, Jinjing Huang, Jingyun Tian, Jingyuan Zhang, Jinquan Sun, Juanhui Tu, Jun Liu, Jun Xu, Jun Zhou, Junjie Ou, Junpeng Fang, Kaihong Zhang, Kaiqin Hu, Ke Shi, Kun Tang, Kunlong Chen, Lanyin Mei, Lei Liang, Lei Xu, Libo Zhang, Lin Ju, Lin Yuan, Ling Zhong, Lintao Ma, Lu Liu, Lu Yu, Lun Cai, Meiqi Zhu, Mengying Li, Min Chen, Minghao Xue, Minghong Cai, Mingming Yin, Peijie Jiang, Peilong Zhao, Pingping Liu, Qian Zhao, Qing Cui, Qingxiang Huang, Qingyuan Yang, Quankun Yu, Shaowei Wei, Shijie Lian, Shoujian Zheng, Shun Song, Shungen Zhang, Shuo Zhang, Siyuan Li, Song Liu, Ting Guo, Tong Zhao, Wanli Gu, Weichang Wu, Weiguang Han, Wenjing Fang, Wubin Wang, Xiang Shu, Xiao Shi, Xiaoshun Lan, Xiaolu Zhang, Xiaqing Sun, Xin Zhao, Xingyu Lu, Xiong Xu, Xudong Wang, Xudong Wang, Xuemin Yang, Yajie Yang, Yang Xiang, Yanzhe Li, Yi Zhang, Yilong Wang, Yingxue Li, Yongzhen Guo, Yuzhuo Fu, Yuanyuan Wang, Yue Yang, Yue Yu, Yufeng Deng, Yun Zhang, Yunfei Yu, Yuqi Zhang, Yuxiao He, Zengke Gui, Zhaoxin Huan, Zhaoyang Wang, Zhibo Zhu, Zhihao Wang, Zhiqiang Zhang, Zhoufei Wang, Zihang Zeng, Ziqi Liu, Zitao Xuan, Zuoli Tang

TL;DR

Ling 2.0 advances scalable general reasoning by unifying a high-sparsity MoE foundation with a principled scaling-law framework, wind-tunnel extrapolation, FP8 training, and a staged pre/post-training regime. The approach demonstrates that sparse activation can deliver both higher reasoning accuracy and lower active compute, establishing a new Pareto frontier at 1T parameters and beyond. Key contributions include Ling Scaling Laws, the Ling Wind Tunnel for efficient innovation, reasoning-focused data pipelines, Evo-CoT RL with LPO, GAR/RubriX human-preference alignment, and an FP8-oriented, co-design-driven infrastructure. Together, these innovations enable more efficient, open, and capable open-language reasoning systems and set a blueprint for future trillion-parameter foundations.

Abstract

We introduce Ling 2.0, a series reasoning-oriented language foundation built upon the principle that every activation boosts reasoning capability. Designed to scale from tens of billions to one trillion parameters under a unified Mixture-of-Experts (MoE) paradigm, Ling 2.0 emphasizes high sparsity, cross-scale consistency, and efficiency guided by empirical scaling laws. The series includes three non-thinking (instruct) models - Ling-mini-2.0, Ling-flash-2.0, and Ling-1T - ranging from 16B to 1T total parameters and achieving up to 7-fold active-compute efficiency compared with dense counterparts. Ling 2.0 integrates coordinated innovations across model architecture, pre-training, post-training, and infrastructure: a high-sparsity MoE with MTP for efficient reasoning, reasoning-oriented data and mid-training CoT activation, reinforcement-based fine-tuning (DFT, Evo-CoT), and full-scale FP8 training with fine-grained heterogeneous pipelines. At the trillion scale, Ling-1T establishes a new Pareto frontier of reasoning accuracy versus computational efficiency, demonstrating that sparse activation, when properly aligned with reasoning objectives, enables scalable and efficient intelligence. Collectively, Ling 2.0 provides a coherent, open, and efficient foundation for advancing future reasoning and thinking models, including the Ring series built upon the same base.

Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation

TL;DR

Ling 2.0 advances scalable general reasoning by unifying a high-sparsity MoE foundation with a principled scaling-law framework, wind-tunnel extrapolation, FP8 training, and a staged pre/post-training regime. The approach demonstrates that sparse activation can deliver both higher reasoning accuracy and lower active compute, establishing a new Pareto frontier at 1T parameters and beyond. Key contributions include Ling Scaling Laws, the Ling Wind Tunnel for efficient innovation, reasoning-focused data pipelines, Evo-CoT RL with LPO, GAR/RubriX human-preference alignment, and an FP8-oriented, co-design-driven infrastructure. Together, these innovations enable more efficient, open, and capable open-language reasoning systems and set a blueprint for future trillion-parameter foundations.

Abstract

We introduce Ling 2.0, a series reasoning-oriented language foundation built upon the principle that every activation boosts reasoning capability. Designed to scale from tens of billions to one trillion parameters under a unified Mixture-of-Experts (MoE) paradigm, Ling 2.0 emphasizes high sparsity, cross-scale consistency, and efficiency guided by empirical scaling laws. The series includes three non-thinking (instruct) models - Ling-mini-2.0, Ling-flash-2.0, and Ling-1T - ranging from 16B to 1T total parameters and achieving up to 7-fold active-compute efficiency compared with dense counterparts. Ling 2.0 integrates coordinated innovations across model architecture, pre-training, post-training, and infrastructure: a high-sparsity MoE with MTP for efficient reasoning, reasoning-oriented data and mid-training CoT activation, reinforcement-based fine-tuning (DFT, Evo-CoT), and full-scale FP8 training with fine-grained heterogeneous pipelines. At the trillion scale, Ling-1T establishes a new Pareto frontier of reasoning accuracy versus computational efficiency, demonstrating that sparse activation, when properly aligned with reasoning objectives, enables scalable and efficient intelligence. Collectively, Ling 2.0 provides a coherent, open, and efficient foundation for advancing future reasoning and thinking models, including the Ring series built upon the same base.
Paper Structure (63 sections, 11 equations, 24 figures, 8 tables)

This paper contains 63 sections, 11 equations, 24 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Architecture
  3. Basic Architecture
  4. Model Optimization
  5. Ling Scaling Laws
  6. Scaling Laws for Optimal Hyper-parameters
  7. Scaling Laws for MoE Architectural Efficiency
  8. Ling Wind Tunnel Experiments for Efficient Innovation
  9. Pre-training
  10. Pre-training Data
  11. General Knowledge Data
  12. Reasoning Data
  13. Ling Code Corpus
  14. Ling Math Corpus
  15. Multilingual Data
  16. ...and 48 more sections

Figures (24)

  • Figure 1: Scaling laws for optimal hyperparameters and optimal model-data allocation. Blue and red lines represent the fitted laws for MoE and dense models, respectively, derived on the same training dataset. Gray circles are the experimental data points used for fitting.
  • Figure 2: Impact of the MoE architectural configuration on loss and efficiency leverage (EL).
  • Figure 3: Illustration of the Ling Wind Tunnel's experimental design (a) and an example analysis (b).
  • Figure 4: Performance of Ling Code Corpus (a) and Ling Math Corpus (b, c).
  • Figure 5: Pre-training and mid-training stages of Ling 2.0. We adopt a multi-stage training strategy that progressively expands the context window from 4K to 128K and introduces reasoning and CoT data in advance to pre-activate the model’s reasoning ability.
  • ...and 19 more figures