Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading

TL;DR

FineMoE tackles the memory waste and latency in MoE-based LLM serving by introducing fine-grained expert offloading. It records per-iteration gate distributions as expert maps and augments them with input semantic embeddings to predict and prefetch needed experts via semantic and trajectory similarity searches, all while operating asynchronously to minimize overhead. The approach achieves a 47% reduction in inference latency and a 39% improvement in expert hit rate over state-of-the-art methods on multiple MoE models and workloads, using a six-GPU testbed and lossless offloading. This work highlights a practical path to scalable, high-throughput MoE serving without sacrificing generation quality, enabling more efficient deployment of large MoE models in real-world settings.

Abstract

Large Language Models (LLMs) have gained immense success in revolutionizing various applications, including content generation, search and recommendation, and AI-assisted operation. To reduce high training costs, Mixture-of-Experts (MoE) architecture has become a popular backbone for modern LLMs. However, despite the benefits, serving MoE-based LLMs experience severe memory inefficiency due to sparsely activated experts. Recent studies propose to offload inactive experts from GPU memory to CPU memory to improve the serving efficiency of MoE models. However, they either incur high inference latency or high model memory footprints due to coarse-grained designs. To tame the latency-memory trade-off in MoE serving, we present FineMoE, a fine-grained expert offloading system for MoE serving that achieves low inference latency with memory efficiency. We design FineMoE to extract fine-grained expert selection patterns from MoE models and semantic hints from input prompts to efficiently guide expert prefetching, caching, and offloading decisions. FineMoE is prototyped on top of HuggingFace Transformers and deployed on a six-GPU testbed. Experiments with open-source MoE models and real-world workloads show that FineMoE reduces inference latency by 47% and improves expert hit rate by 39% over state-of-the-art solutions.

Figures (18)

• Figure 1: Mixture-of-Experts (MoE) Large Language Model (LLM) serving.
• Figure 2: The design space of MoE-based LLM serving.
• Figure 3: Expert pattern and predictability analysis in coarse granularity (request-level) and fine granularity (iteration-level).
• Figure 4: Expert hit rates of coarse-grained and fine-grained expert offloading designs when serving Mixtral-8$\times$7B, Qwen1.5-MoE, and Phi-3.5-MoE with LMSYS-Chat-1M at different prefetch distances, respectively.
• Figure 5: FineMoE's architecture and workflow.