Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models

TL;DR

The work tackles memory-efficient deployment of Mixture-of-Experts by quantifying local routing consistency with two metrics, SRP and SCH. It analyzes 20 MoE LLMs to reveal how locality interacts with local and global load balancing, and how domain-specialized experts enhance consistency more than vocabulary-specialized ones, with shared experts potentially harming locality. A practical takeaway is that cache sizes around twice the number of active experts achieve favorable hit rates and throughput, informing MoE offloading designs. The authors also provide open-source code to replicate and build on their findings.

Abstract

Mixture-of-Experts (MoE) enables efficient scaling of large language models (LLMs) with sparsely activated experts during inference. To effectively deploy large MoE models on memory-constrained devices, many systems introduce *expert offloading* that caches a subset of experts in fast memory, leaving others on slow memory to run on CPU or load on demand. While some research has exploited the locality of expert activations, where consecutive tokens activate similar experts, the degree of this **local routing consistency** varies across models and remains understudied. In this paper, we propose two metrics to measure local routing consistency of MoE models: (1) **Segment Routing Best Performance (SRP)**, which evaluates how well a fixed group of experts can cover the needs of a segment of tokens, and (2) **Segment Cache Best Hit Rate (SCH)**, which measures the hit rate of an expert cache utilizing a length of future information under a cache limit. We analyze 20 MoE LLMs with diverse sizes and architectures and use toy models to verify key factors related to local routing consistency. We find a strong trade-off between local routing consistency and *local* load balance, while showing that *global* load balance can coexist with local routing consistency. Meanwhile, settings like shared experts that decrease expert combination space can lead to low local routing consistency. We further reveal that domain-specialized experts contribute more to routing consistency than vocabulary-specialized ones, and that most models balance between cache effectiveness and efficiency with cache sizes approximately twice the active experts. These findings pave the way for memory-efficient MoE design and deployment without compromising inference speed. We publish the code for replicating experiments at https://github.com/ljcleo/moe-lrc .