Sigma-Moe-Tiny Technical Report

TL;DR

<3-5 sentence high-level summary> Sigma-MoE-Tiny pushes Mixture-of-Experts sparsity to an extreme, constructing a 20B-parameter decoder-only Transformer where only 0.5B parameters are activated per token by routing to a single expert among up to 96 per layer. The authors address severe load-balancing challenges with a progressive sparsification schedule, stabilized pre-training on a diverse corpus, and a multi-stage post-training curriculum that extends context length and enhances reasoning. Empirical results show strong performance across general, mathematics, and coding benchmarks at small active parameter budgets, with post-training matching or exceeding larger dense and MoE models. The work also contributes analytical insights into load balancing in highly sparse MoE architectures and outlines practical directions for future sparse MoE design.

Abstract

Mixture-of-Experts (MoE) has emerged as a promising paradigm for foundation models due to its efficient and powerful scalability. In this work, we present Sigma-MoE-Tiny, an MoE language model that achieves the highest sparsity compared to existing open-source models. Sigma-MoE-Tiny employs fine-grained expert segmentation with up to 96 experts per layer, while activating only one expert for each token, resulting in 20B total parameters with just 0.5B activated. The major challenge introduced by such extreme sparsity lies in expert load balancing. We find that the widely-used load balancing loss tends to become ineffective in the lower layers under this setting. To address this issue, we propose a progressive sparsification schedule aiming to balance expert utilization and training stability. Sigma-MoE-Tiny is pre-trained on a diverse and high-quality corpus, followed by post-training to further unlock its capabilities. The entire training process remains remarkably stable, with no occurrence of irrecoverable loss spikes. Comprehensive evaluations reveal that, despite activating only 0.5B parameters, Sigma-MoE-Tiny achieves top-tier performance among counterparts of comparable or significantly larger scale. In addition, we provide an in-depth discussion of load balancing in highly sparse MoE models, offering insights for advancing sparsity in future MoE architectures. Project page: https://qghuxmu.github.io/Sigma-MoE-Tiny Code: https://github.com/microsoft/ltp-megatron-lm

Figures (6)

• Figure 1: Left: GPQA-Diamond accuracy vs. activated parameters across different open-source LLMs, demonstrating that Sigma-MoE-Tiny achieves advanced capability with only 0.5B activated parameters. Right: The trend of sparsity in mainstream MoE models over time is shown. Here, sparsity is defined as the ratio of total to activated parameters. With a total-to-activated ratio of 40:1, Sigma-MoE-Tiny achieves the highest sparsity among existing open-source models.
• Figure 1: Model architecture of Sigma-MoE-Tiny.
• Figure 2: (a) Relative deviation from uniform token allocation is defined as the ratio between the difference of an expert's actual token count and the ideal uniform count, normalized by the uniform count. We report this deviation for the max-loaded and min-loaded experts in Layer 0. (b) and (c) show the distribution of token allocation fraction $f$ and gating probability $p$ across all experts in Layer 0 and Layer 52, respectively.
• Figure 3: Relative deviation from uniform token allocation for the max-loaded and min-loaded experts in Layer 0. Introducing loss-free balancing strategy substantially aggravates load imbalance under the setting of 96 experts with 1 activated.
• Figure 4: Comparison of MMLU performance between Top-1 LBL and conventional LBL under different training token counts.