LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing

TL;DR

The problems of scaling MoE training in existing systems are presented and the potential of exploiting token similarity to facilitate data compression is highlighted and an efficient LSH-based compression technique is introduced, which utilizes the cross-polytope hashing for rapid clustering and implements a residual-based error compensation scheme to alleviate the adverse impact of compression.

Abstract

Larger transformer models always perform better on various tasks but require more costs to scale up the model size. To efficiently enlarge models, the mixture-of-experts (MoE) architecture is widely adopted, which consists of a gate network and a series of experts and keep the training cost constant by routing the input data to a fixed number of experts instead of all. In existing large-scale MoE training systems, experts would be distributed among different GPUs for parallelization, and thus input data requires additional all-to-all communications to access the target experts and conduct corresponding computations. However, upon evaluating the training process of three mainstream MoE models on commonly used GPU clusters, we found that the all-to-all communication ratio averaged around 45%, which significantly hinders the efficiency and scalability of training MoE models. In this paper, we propose LSH-MoE, a communication-efficient MoE training framework using locality-sensitive hashing (LSH). We first present the problems of scaling MoE training in existing systems and highlight the potential of exploiting token similarity to facilitate data compression. Then, we introduce an efficient LSH-based compression technique, which utilizes the cross-polytope hashing for rapid clustering and implements a residual-based error compensation scheme to alleviate the adverse impact of compression. To verify the effectiveness of our methods, we conduct experiments on both language models (e.g., RoBERTa, GPT, and T5) and vision models (e.g., Swin) for pre-training and fine-tuning tasks. The results demonstrate that our method substantially outperforms its counterparts across different tasks by 1.28x - 2.2x of speedup.

Figures (7)

• Figure 1: Mixture-of-Experts on a single GPU.
• Figure 2: Training Mixture-of-Experts on multiple GPUs as expert parallelism.
• Figure 3: Proportion of all-to-all communication time relative to total training duration across different configurations: scaling the number of training servers (Figure \ref{['fig:a2a_percent_32gpus']}) and scaling the parameter size of models (Figure \ref{['fig:a2a_percent_more_experts']}).
• Figure 4: Principal Component Analysis (PCA) Visualization of input tokens involved in all-to-all communication.
• Figure 5: Schematic of MoE training with Locality-Sensitive Hashing (LSH-MoE).