Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate

TL;DR

Fate tackles the memory and I/O bottlenecks of deploying sparse Mixture-of-Experts (MoE) models on edge devices. It introduces cross-layer prefetching that uses adjacent gate inputs to predict which experts to load, and a shallow-favoring cache managed by Adaptive Replacement Cache, combined with per-expert quantization to reduce I/O and memory usage. The approach yields substantial speedups in both prefill and decoding stages across different memory budgets while maintaining accuracy, with up to 4.5× (prefill) and 4.1× (decoding) improvements over baselines. This work demonstrates that fast, edge-friendly MoE inference is feasible without extensive retraining, enabling privacy-aware and responsive on-device LLM applications.

Abstract

Large Language Models (LLMs) have demonstrated impressive performance across various tasks, and their application in edge scenarios has attracted significant attention. However, sparse-activated Mixture-of-Experts (MoE) models, which are well suited for edge scenarios, have received relatively little attention due to their high memory demands. Offload-based methods have been proposed to address this challenge, but they face difficulties with expert prediction. Inaccurate expert predictions can result in prolonged inference delays. To promote the application of MoE models in edge scenarios, we propose Fate, an offloading system designed for MoE models to enable efficient inference in resource-constrained environments. The key insight behind Fate is that gate inputs from adjacent layers can be effectively used for expert prefetching, achieving high prediction accuracy without additional GPU overhead. Furthermore, Fate employs a shallow-favoring expert caching strategy that increases the expert hit rate to 99\%. Additionally, Fate integrates tailored quantization strategies for cache optimization and IO efficiency. Experimental results show that, compared to Load on Demand and Expert Activation Path-based method, Fate achieves up to 4.5x and 1.9x speedups in prefill speed and up to 4.1x and 2.2x speedups in decoding speed, respectively, while maintaining inference quality. Moreover, Fate's performance improvements are scalable across different memory budgets.

Figures (14)

• Figure 1: Comparison of three kinds of pipeline.
• Figure 2: Two mainstream structural designs of MoE model.
• Figure 3: Overview of Fate.
• Figure 4: Comparison between true and predicted routing weights at block $i+1$.
• Figure 5: Workflow of cross-layer expert prefetch.