Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training Report of TeleChat3-MoE

Xinzhang Liu, Chao Wang, Zhihao Yang, Zhuo Jiang, Xuncheng Zhao, Haoran Wang, Lei Li, Dongdong He, Luobin Liu, Kaizhe Yuan, Han Gao, Zihan Wang, Yitong Yao, Sishi Xiong, Wenmin Deng, Haowei He, Kaidong Yu, Yu Zhao, Ruiyu Fang, Yuhao Jiang, Yingyan Li, Xiaohui Hu, Xi Yu, Jingqi Li, Yanwei Liu, Qingli Li, Xinyu Shi, Junhao Niu, Chengnuo Huang, Yao Xiao, Ruiwen Wang, Fengkai Li, Luwen Pu, Kaipeng Jia, Fubei Yao, Yuyao Huang, Xuewei He, Zhuoru Jiang, Ruiting Song, Rui Xue, Qiyi Xie, Jie Zhang, Zilu Huang, Zhaoxi Zhang, Zhilong Lu, Yanhan Zhang, Yin Zhang, Yanlei Xue, Zhu Yuan, Teng Su, Xin Jiang, Shuangyong Song, Yongxiang Li, Xuelong Li

TL;DR

TeleChat3-MoE tackles the challenge of training frontier-scale Mixture-of-Experts language models on Ascend NPUs by delivering a comprehensive, end-to-end training stack. The paper combines rigorous numerical accuracy verification (operator-level and end-to-end), hardware-aware architectural design (MLA, shallow-and-wide MoE), and extensive performance optimizations (interleaved pipelines, long-sequence scheduling, hierarchical EP communication, and DVM-based fusion) with an automated ILP-enabled parallelization framework to explore and optimize complex multi-dimensional parallelism. It demonstrates near-linear scaling on clusters with thousands of NPUs and provides methodology for stable cross-hardware migrations and parallelism changes, supported by practical deployment cases and open-source releases. Collectively, these contributions establish a mature, scalable, full-stack solution for large-scale MoE model development on domestic hardware ecosystems, enabling efficient exploration of models at trillion-parameter scales.

Abstract

TeleChat3-MoE is the latest series of TeleChat large language models, featuring a Mixture-of-Experts (MoE) architecture with parameter counts ranging from 105 billion to over one trillion,trained end-to-end on Ascend NPU cluster. This technical report mainly presents the underlying training infrastructure that enables reliable and efficient scaling to frontier model sizes. We detail systematic methodologies for operator-level and end-to-end numerical accuracy verification, ensuring consistency across hardware platforms and distributed parallelism strategies. Furthermore, we introduce a suite of performance optimizations, including interleaved pipeline scheduling, attention-aware data scheduling for long-sequence training,hierarchical and overlapped communication for expert parallelism, and DVM-based operator fusion. A systematic parallelization framework, leveraging analytical estimation and integer linear programming, is also proposed to optimize multi-dimensional parallelism configurations. Additionally, we present methodological approaches to cluster-level optimizations, addressing host- and device-bound bottlenecks during large-scale training tasks. These infrastructure advancements yield significant throughput improvements and near-linear scaling on clusters comprising thousands of devices, providing a robust foundation for large-scale language model development on hardware ecosystems.

Training Report of TeleChat3-MoE

TL;DR

TeleChat3-MoE tackles the challenge of training frontier-scale Mixture-of-Experts language models on Ascend NPUs by delivering a comprehensive, end-to-end training stack. The paper combines rigorous numerical accuracy verification (operator-level and end-to-end), hardware-aware architectural design (MLA, shallow-and-wide MoE), and extensive performance optimizations (interleaved pipelines, long-sequence scheduling, hierarchical EP communication, and DVM-based fusion) with an automated ILP-enabled parallelization framework to explore and optimize complex multi-dimensional parallelism. It demonstrates near-linear scaling on clusters with thousands of NPUs and provides methodology for stable cross-hardware migrations and parallelism changes, supported by practical deployment cases and open-source releases. Collectively, these contributions establish a mature, scalable, full-stack solution for large-scale MoE model development on domestic hardware ecosystems, enabling efficient exploration of models at trillion-parameter scales.

Abstract

TeleChat3-MoE is the latest series of TeleChat large language models, featuring a Mixture-of-Experts (MoE) architecture with parameter counts ranging from 105 billion to over one trillion,trained end-to-end on Ascend NPU cluster. This technical report mainly presents the underlying training infrastructure that enables reliable and efficient scaling to frontier model sizes. We detail systematic methodologies for operator-level and end-to-end numerical accuracy verification, ensuring consistency across hardware platforms and distributed parallelism strategies. Furthermore, we introduce a suite of performance optimizations, including interleaved pipeline scheduling, attention-aware data scheduling for long-sequence training,hierarchical and overlapped communication for expert parallelism, and DVM-based operator fusion. A systematic parallelization framework, leveraging analytical estimation and integer linear programming, is also proposed to optimize multi-dimensional parallelism configurations. Additionally, we present methodological approaches to cluster-level optimizations, addressing host- and device-bound bottlenecks during large-scale training tasks. These infrastructure advancements yield significant throughput improvements and near-linear scaling on clusters comprising thousands of devices, providing a robust foundation for large-scale language model development on hardware ecosystems.
Paper Structure (18 sections, 9 figures, 5 tables)

This paper contains 18 sections, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Model Architecture
  3. Accuracy Verification
  4. Operator-wise Accuracy Verification
  5. End-to-End Model Accuracy Alignment
  6. Cross-Hardware Precision Alignment
  7. Numerical Equivalence Validation Across Parallelism Strategies
  8. Training Framework Performance Optimization
  9. Interleaved Pipeline Scheduling with Communication overlapping
  10. Long-Sequence Optimization via Attention-Aware Scheduling
  11. Hierarchical Communication Merging for Expert Parallelism
  12. Communication Overlapping for Expert Parallelism
  13. DVM-Based Operator Fusion
  14. Conventional Expert-driven Tuning Challenges
  15. Parallelization Strategy Generation
  16. ...and 3 more sections

Figures (9)

  • Figure 1: Workflow for cross-hardware training precision alignment.
  • Figure 2: Interleaved pipeline scheduling with 1F1B overlap in MindSpore.
  • Figure 3: Attention-aware data scheduling for load balancing in long-sequence sparse attention training.
  • Figure 4: Hierarchical communication scheme for expert parallelism, reducing redundant cross-machine traffic.
  • Figure 5: Expert parallelism communication overlapping via multi-dimensional data partitioning and fine-grained scheduling.
  • ...and 4 more figures