MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs

TL;DR

MoE-Prism addresses the rigidity of MoE serving, where the top-$k$ gating over a small number of monolithic experts creates a coarse cost–quality spectrum. It introduces a model-system co-design with an offline Refactoring Engine that decomposes each expert into finer sub-experts and a lightweight gating mechanism, and an online Scheduling Engine that optimizes request batching and quality level under QoS constraints. The approach yields over 4× more distinct operating points and up to 19.9% throughput gains under strict latency budgets, while reducing latency by up to 10.36% on memory-constrained devices. This work enables true elastic MoE serving, offering finer-grained control, improved resource utilization, and adaptability to heterogeneous SLOs in real-world deployments.

Abstract

Mixture-of-Experts (MoE) models, the state-of-the-art in large-scale AI, achieve high quality by sparsely activating parameters. However, their reliance on routing between a few monolithic experts via a top-k mechanism creates a "quality cliff", offering only a few coarse-grained operating points. This inflexibility forces a difficult trade-off between cost and quality, preventing adaptation to diverse Service Level Objectives (SLOs) and leading to significant resource over-provisioning. This paper introduces MoE-Prism, a model-system co-design that transforms rigid MoE models into elastic services. Our methodology is divided into two phases. First, an \emph{Offline Refactoring Engine} systematically deconstructs monolithic experts into fine-grained "sub-experts." This engine employs a partitioning optimization solver that uses a metaheuristic-based approach to group neurons, preserving functional locality without requiring retraining. Second, an \emph{Online Scheduling Engine} leverages this new elasticity through QoS-aware scheduling. It implements specialized policies to solve complex system problems, including maximizing throughput in cloud deployments and managing latency-optimized offloading for memory-constrained devices. Our evaluation across three different MoE models shows that MoE-Prismprovides over 4 times more distinct, stable operating points than the baseline. This allows an AI service to dynamically improve throughput by up to 19.9\% under a strict latency budget or reduce latency by up to 10.36\% under limited resources. MoE-Prism provides the critical "control knob" to bridge the model-system gap, enabling the next generation of adaptive, efficient, and QoS-aware AI services.

Figures (12)

• Figure 1: MoE-Prism resolves the Quality Cliff in MoE serving.(Left) Conventional MoE models are rigid, creating a "Quality Cliff" where achieving a target speedup forces a disproportionately large drop in model quality. The name reflects its core function: just as a prism decomposes a single beam of light into a spectrum of colors, MoE-Prism refactors a monolithic expert into a spectrum of fine-grained sub-experts. This introduces architectural elasticity (right), transforming the cliff into a smooth trade-off curve and enabling the selection of an optimal "Target Quality" point that was previously unattainable.
• Figure 2: Computational flow of a Mixture-of-Experts (MoE) layer. For each token, the gating network selects the top-k experts (here, $k=2$) from a large bank of $N$ experts. The final output is the weighted sum of the outputs from only the activated experts.
• Figure 3: Distribution of neuron activation magnitudes within a single activated expert for a sample of input tokens. A vast majority of neurons exhibit near-zero activation, demonstrating significant computational redundancy. This suggests that only a small fraction of the expert's computation is essential for any given token.
• Figure 4: The MoE-Prism System Architecture. The co-design consists of two phases. The Offline Refactoring Engine performs a one-time transformation of a standard MoE model, deconstructing monolithic experts into fine-grained sub-experts. The Online Scheduling Engine then leverages this elasticity to power sophisticated, QoS-aware serving strategies, such as offloading sub-experts to overcome GPU memory limits or co-optimizing batching and quality selection to maximize server throughput.
• Figure 5: The partitioning process leverages the neuron-level independence of the FFN to group neurons into new sub-experts. The total expert output is the sum of the outputs of its constituent sub-experts.