Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs

Yichun Yin, Wenyong Huang, Kaikai Song, Yehui Tang, Xueyu Wu, Wei Guo, Peng Guo, Yaoyuan Wang, Xiaojun Meng, Yasheng Wang, Dong Li, Can Chen, Dandan Tu, Yin Li, Fisher Yu, Ruiming Tang, Yunhe Wang, Baojun Wang, Bin Wang, Bo Wang, Boxiao Liu, Changzheng Zhang, Duyu Tang, Fei Mi, Hui Jin, Jiansheng Wei, Jiarui Qin, Jinpeng Li, Jun Zhao, Liqun Deng, Lin Li, Minghui Xu, Naifu Zhang, Nianzu Zheng, Qiang Li, Rongju Ruan, Shengjun Cheng, Tianyu Guo, Wei He, Wei Li, Weiwen Liu, Wulong Liu, Xinyi Dai, Yonghan Dong, Yu Pan, Yue Li, Yufei Wang, Yujun Li, Yunsheng Ni, Zhe Liu, Zhenhe Zhang, Zhicheng Liu

TL;DR

Pangu Ultra tackles the challenge of training a very large dense LLM on Ascend NPUs by addressing stability and efficiency. It introduces the depth-scaled sandwich norm and TinyInit to stabilize gradients in deep dense transformers, paired with a domain-aware tokenizer and a three-phase pre-training regimen that includes long-context extension to 128K tokens. Post-training aligns the model with human preferences via SFT and RL, yielding state-of-the-art results on many English and Chinese benchmarks and strong reasoning performance on targeted tasks. The work demonstrates practical viability of dense models at this scale on Ascend hardware, achieving substantial hardware utilization (MFU) and providing a blueprint for scalable, efficient dense-LMM training with industrial-grade accelerators.

Abstract

We present Pangu Ultra, a Large Language Model (LLM) with 135 billion parameters and dense Transformer modules trained on Ascend Neural Processing Units (NPUs). Although the field of LLM has been witnessing unprecedented advances in pushing the scale and capability of LLM in recent years, training such a large-scale model still involves significant optimization and system challenges. To stabilize the training process, we propose depth-scaled sandwich normalization, which effectively eliminates loss spikes during the training process of deep models. We pre-train our model on 13.2 trillion diverse and high-quality tokens and further enhance its reasoning capabilities during post-training. To perform such large-scale training efficiently, we utilize 8,192 Ascend NPUs with a series of system optimizations. Evaluations on multiple diverse benchmarks indicate that Pangu Ultra significantly advances the state-of-the-art capabilities of dense LLMs such as Llama 405B and Mistral Large 2, and even achieves competitive results with DeepSeek-R1, whose sparse model structure contains much more parameters. Our exploration demonstrates that Ascend NPUs are capable of efficiently and effectively training dense models with more than 100 billion parameters. Our model and system will be available for our commercial customers.

Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs

TL;DR

Pangu Ultra tackles the challenge of training a very large dense LLM on Ascend NPUs by addressing stability and efficiency. It introduces the depth-scaled sandwich norm and TinyInit to stabilize gradients in deep dense transformers, paired with a domain-aware tokenizer and a three-phase pre-training regimen that includes long-context extension to 128K tokens. Post-training aligns the model with human preferences via SFT and RL, yielding state-of-the-art results on many English and Chinese benchmarks and strong reasoning performance on targeted tasks. The work demonstrates practical viability of dense models at this scale on Ascend hardware, achieving substantial hardware utilization (MFU) and providing a blueprint for scalable, efficient dense-LMM training with industrial-grade accelerators.

Abstract

We present Pangu Ultra, a Large Language Model (LLM) with 135 billion parameters and dense Transformer modules trained on Ascend Neural Processing Units (NPUs). Although the field of LLM has been witnessing unprecedented advances in pushing the scale and capability of LLM in recent years, training such a large-scale model still involves significant optimization and system challenges. To stabilize the training process, we propose depth-scaled sandwich normalization, which effectively eliminates loss spikes during the training process of deep models. We pre-train our model on 13.2 trillion diverse and high-quality tokens and further enhance its reasoning capabilities during post-training. To perform such large-scale training efficiently, we utilize 8,192 Ascend NPUs with a series of system optimizations. Evaluations on multiple diverse benchmarks indicate that Pangu Ultra significantly advances the state-of-the-art capabilities of dense LLMs such as Llama 405B and Mistral Large 2, and even achieves competitive results with DeepSeek-R1, whose sparse model structure contains much more parameters. Our exploration demonstrates that Ascend NPUs are capable of efficiently and effectively training dense models with more than 100 billion parameters. Our model and system will be available for our commercial customers.

Paper Structure

This paper contains 39 sections, 1 equation, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Model Architecture
  3. Depth-Scaled Sandwich-Norm
  4. Model Initialization
  5. Tokenizer
  6. Model Training
  7. Pre-training Stage
  8. Data Construction
  9. Data Quality Assessment
  10. Pre-training Parameters
  11. Long Context Extension
  12. Post-training Alignment
  13. Post-training Data
  14. Post-training Strategy
  15. Training System
  16. ...and 24 more sections

Figures (10)

  • Figure 1: Structure comparison between Pre-Layer Norm (Pre-LN) and Depth-Scaled Sandwich-Norm (DSSN). DSSN applies normalization layers to both before and after the attention and FFN block, while Pre-LN only utilizes one normalization layer. DSSN also employs a depth-scaled initialization schema, which is not in the original sandwich norm.
  • Figure 2: Pipeline parallelism and the interleaved pipeline-parallel scheduling.
  • Figure 3: A Comparison of the default transformer computation and the MC2 method. Note that in actual training, communication and computation tasks are fused into a single kernel in MC2.
  • Figure 4: Examples of attention mask compression for the NFA operator.
  • Figure 5: Examples of the mechanism of sub-sequence partitioning for context parallelism.
  • ...and 5 more figures