MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts

TL;DR

MoBiLE tackles the CPU-GPU bandwidth bottleneck in offloading-based MoE inference on consumer GPUs by using a mixture of big and little experts. It activates $K/2$ experts for the majority of tokens and resorts to a full $K$-expert fallback for important tokens, guided by a training-free prefetching mechanism that overlaps memory I/O and computation. The approach is plug-and-play and requires no model retraining, leveraging logits from the little model to prefetch the big-model experts and employing a threshold-driven fallback ($\gamma$) to maintain accuracy. Evaluations on contemporary MoE models demonstrate end-to-end speedups of roughly $1.60$–$1.72\times$ with negligible accuracy loss, enabling efficient MoE inference on consumer hardware such as RTX 4080-class GPUs.

Abstract

Mixture-of-Experts (MoE) models have recently demonstrated exceptional performance across a diverse range of applications. The principle of sparse activation in MoE models facilitates an offloading strategy, wherein active experts are maintained in GPU HBM, while inactive experts are stored in CPU DRAM. The efficacy of this approach, however, is fundamentally constrained by the limited bandwidth of the CPU-GPU interconnect. To mitigate this bottleneck, existing approaches have employed prefetching to accelerate MoE inference. These methods attempt to predict and prefetch the required experts using specially trained modules. Nevertheless, such techniques are often encumbered by significant training overhead and have shown diminished effectiveness on recent MoE models with fine-grained expert segmentation. In this paper, we propose MoBiLE, a plug-and-play offloading-based MoE inference framework with \textit{mixture of big-little experts}. It reduces the number of experts for unimportant tokens to half for acceleration while maintaining full experts for important tokens to guarantee model quality. Further, a dedicated fallback and prefetching mechanism is designed for switching between little and big experts to improve memory efficiency. We evaluate MoBiLE on four typical modern MoE architectures and challenging generative tasks. Our results show that MoBiLE achieves a speedup of 1.60x to 1.72x compared to the baseline on a consumer GPU system, with negligible degradation in accuracy.

Figures (5)

• Figure 1: Upper left: Illustration of the MoE module. The router dynamically determines which experts to activate based on the current input. Upper right: parameter statistics of modern SOTA MoE models. Lower: Offloading-based inference for MoE models, where non-activated experts are stored in CPU DRAM, and the activated parameters are transferred to the GPU HBM for computation.
• Figure 2: For MoE models, activating less than the default number of activated experts leads to no difference for most of the tokens (unimportant tokens), while a small amount of important tokens need more experts to process.
• Figure 3: Left: Latency comparison of different methods, including full MoE, MoBiLE little expert, and MoBiLE big expert. Right: comparison of different inference schemes. Traditional MoE inference suffers from high latency, while naively reducing the number of activated experts leads to a high accuracy decrease. MoBiLE maintains high accuracy while achieving low latency.
• Figure 4: Comparison of pre-gated MoE prefetch and MoBiLE prefech.
• Figure 5: Left: Accuracy-efficiency tradeoff for MoBiLE. Right: Speedup of MoBiLE compared with SOTA MoE offloading methods.