MoEless: Efficient MoE LLM Serving via Serverless Computing

TL;DR

MoEless is presented, the first serverless MoE serving framework that mitigates expert load imbalance and accelerates inference via serverless experts, and design optimized expert scaling and placement strategies to maximize function locality, improve GPU utilization, and balance loads across experts and GPUs.

Abstract

Large Language Models (LLMs) have become a cornerstone of AI, driving progress across diverse domains such as content creation, search and recommendation systems, and AI-assisted workflows. To alleviate extreme training costs and advancing model scales, Mixture-of-Experts (MoE) has become a popular backbone for modern LLMs, which are commonly served in distributed deployment using expert parallelism (EP). However, MoE's sparse activation mechanism leads to severe expert load imbalance, where a few experts become overloaded while others remain idle, resulting in expert stragglers that inflate inference latency and serving cost. Existing expert load balancing solutions assume static resource configurations on serverful infrastructures, limiting expert scalability and elasticity, and resulting in either costly real-time expert swapping or degraded generation quality. We present MoEless, the first serverless MoE serving framework that mitigates expert load imbalance and accelerates inference via serverless experts. MoEless employs lightweight, layer-aware predictors to accurately estimate incoming expert load distributions and proactively identify stragglers. We design optimized expert scaling and placement strategies to maximize function locality, improve GPU utilization, and balance loads across experts and GPUs. MoEless is prototyped on top of Megatron-LM and deployed on an eight-GPU testbed. Experiments with open-source MoE models and real-world workloads show that MoEless reduces inference latency by 43% and inference cost by 84% compared to state-of-the-art solutions.

Figures (17)

• Figure 1: Expert load imbalance across layers for different MoE models and datasets: (a) Mixtral-8$\times$7B on ShareGPT and (b) Phi-3.5-MoE on LMSYS-Chat-1M.
• Figure 2: Illustration of serving Mixture-of-Experts (MoE) based Large Language Models under expert parallelism, where tokens are routed by per-layer gate networks to a sparse set of experts distributed across GPUs. Expert load imbalance triggers inefficient resource provisioning ( e.g., over-scaling hot experts or under-utilizing cold ones), thereby increasing serving cost. Embed: embedding layer, TB: Transformer Block, Head: language modeling head, Attention: attention layer, Gate: gate networks, DP: data parallelism, MP: model parallelism.
• Figure 3: Serving Phi-3.5-MoE on LMSYS-Chat-1M using Azure LLM traces: (a) request arrivals, (b) aggregated token loads, and (c) total number of active experts.
• Figure 4: Inference performance of three approaches when serving Phi-3.5-MoE on ShareGPT.
• Figure 5: The architecture and workflow of MoEless.