MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models

TL;DR

MoE-DisCo addresses the prohibitive cost of training large Mixture-of-Experts models by decoupling the full MoE into multiple dense submodels (shared backbone + single expert) trained independently on low-cost hardware, guided by unsupervised clustering to assign semantically coherent data to each expert. After independent submodel training, experts and shared parameters are reintegrated with a weighted parameter fusion and a brief global fine-tune on high-memory GPUs to recover coordinated gating. Empirically, MoE-DisCo achieves comparable or better language modeling and downstream task performance while reducing total training cost by up to ~70% and accelerating convergence across Qwen and LLaMA MoE architectures. The approach leverages Block Coordinate Descent and SimulParallel SGD principles, enabling scalable, hardware-heterogeneous MoE training with minimal inter-device communication and practical applicability for resource-constrained teams.

Abstract

Training large-scale Mixture-of-Experts (MoE) models typically requires high-memory, high-bandwidth GPUs (e.g., A100), and their high cost has become a major barrier to large-model training. In contrast, affordable hardware is low-cost but constrained by memory capacity and bandwidth, making it unsuitable for direct LLM training. To address this, we propose MoE-DisCo (Mixture-of-Experts with Disentangled Clustering and Coordination), a staged training framework. MoE-DisCo decomposes the MoE model into multiple dense submodels, each consisting of a shared backbone and a single expert, and partitions the training data into subsets using unsupervised clustering. Each submodel is trained independently and in parallel on its assigned data subset using low-cost devices, without any inter-device communication. Subsequently, all experts are integrated into a complete MoE model and fine-tuned globally for a short period on high-memory, high-bandwidth GPUs. Experiments show that our method matches or even surpasses full-parameter training in performance across multiple downstream tasks, loss function, and perplexity (PPL), while reducing training cost by 47.6 percent to 69.5 percent on Qwen1.5-MoE-2.7B and Llama-MoE-3.5B across different datasets.

Figures (8)

• Figure 1: Model FLOPs Utilization (MFU) per GPU under 3D parallelism for language models of different scales as the number of GPUs increases. As the cluster scales from tens to thousands of GPUs, MFU per GPU consistently declines across all model sizes.DBLP:journals/corr/abs-2104-04473
• Figure 2: Comparison of training cost profiles between MoE-DisCo (left) and traditional MoE training (right). MoE-DisCo first performs the majority of training on low-cost GPUs (cost $c_0$), which can be highly parallelized across multiple devices, followed by a short fine-tune phase on high-cost GPUs (cost $c_1$). In contrast, traditional methods rely solely on high-cost hardware throughout the entire training process. The total cost (shaded area under the curve) is significantly reduced in MoE-DisCo, demonstrating its efficiency in lowering the monetary burden of large-scale MoE training.
• Figure 3: Overview of the MoE-DisCo training framework. The original MoE model is decomposed into multiple dense submodels, each containing a single expert and shared parameters. Training data is clustered via k-means to assign semantically distinct subsets to individual experts. Submodels are trained independently on low-cost GPUs, enabling high parallelism and reduced hardware cost. Finally, the experts are integrated into a unified MoE architecture and fine-tuned on a high-cost GPU using the full dataset.
• Figure 4: Loss trends between MoE-Disco on fine-tune stage and Full-Parameter across different datasets
• Figure 5: PPL trends between MoE-Disco on fine-tune stage and Full-Parameter across different datasets