Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices

Byeongju Kim, Jungwan Lee, Donghyeon Han, Hoi-Jun Yoo, Sangyeob Kim

TL;DR

This work addresses the memory bottleneck of large-scaleMixture-of-Experts inference on edge devices by offloading inactive experts to SSD and loading on-demand, enabled by a lightweight, per-layer cache. It introduces FlashMoE, which separates experts from non-experts, loads non-experts at initialization, and uses an ML-based cache policy that combines recency and frequency to predict eviction decisions. The approach yields higher cache hit rates and lower I/O, translating to substantial end-to-end latency improvements: up to 4× faster initial loading and up to 2.6× overall speedups on real hardware across models like OLMoE-1B-7B and Qwen3-30B-A3B. The results demonstrate the practicality of running large sparse MoEs on memory-limited devices, expanding feasible on-device AI deployments.

Abstract

Recently, Mixture-of-Experts (MoE) models have gained attention for efficiently scaling large language models. Although these models are extremely large, their sparse activation enables inference to be performed by accessing only a fraction of the model at a time. This property opens the possibility of on-device inference of MoE, which was previously considered infeasible for such large models. Consequently, various systems have been proposed to leverage this sparsity and enable efficient MoE inference for edge devices. However, previous MoE inference systems like Fiddler[8] or DAOP[13] rely on DRAM-based offloading and are not suitable for memory constrained on-device environments. As recent MoE models grow to hundreds of gigabytes, RAM-offloading solutions become impractical. To address this, we propose FlashMoE, a system that offloads inactive experts to SSD, enabling efficient MoE inference under limited RAM. FlashMoE incorporates a lightweight ML-based caching strategy that adaptively combines recency and frequency signals to maximize expert reuse, significantly reducing storage I/O. In addition, we built a user-grade desktop platform to demonstrate the practicality of FlashMoE. On this real hardware setup, FlashMoE improves cache hit rate by up to 51% over well-known offloading policies such as LRU and LFU, and achieves up to 2.6x speedup compared to existing MoE inference systems.

FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices

TL;DR

This work addresses the memory bottleneck of large-scaleMixture-of-Experts inference on edge devices by offloading inactive experts to SSD and loading on-demand, enabled by a lightweight, per-layer cache. It introduces FlashMoE, which separates experts from non-experts, loads non-experts at initialization, and uses an ML-based cache policy that combines recency and frequency to predict eviction decisions. The approach yields higher cache hit rates and lower I/O, translating to substantial end-to-end latency improvements: up to 4× faster initial loading and up to 2.6× overall speedups on real hardware across models like OLMoE-1B-7B and Qwen3-30B-A3B. The results demonstrate the practicality of running large sparse MoEs on memory-limited devices, expanding feasible on-device AI deployments.

Abstract

Recently, Mixture-of-Experts (MoE) models have gained attention for efficiently scaling large language models. Although these models are extremely large, their sparse activation enables inference to be performed by accessing only a fraction of the model at a time. This property opens the possibility of on-device inference of MoE, which was previously considered infeasible for such large models. Consequently, various systems have been proposed to leverage this sparsity and enable efficient MoE inference for edge devices. However, previous MoE inference systems like Fiddler[8] or DAOP[13] rely on DRAM-based offloading and are not suitable for memory constrained on-device environments. As recent MoE models grow to hundreds of gigabytes, RAM-offloading solutions become impractical. To address this, we propose FlashMoE, a system that offloads inactive experts to SSD, enabling efficient MoE inference under limited RAM. FlashMoE incorporates a lightweight ML-based caching strategy that adaptively combines recency and frequency signals to maximize expert reuse, significantly reducing storage I/O. In addition, we built a user-grade desktop platform to demonstrate the practicality of FlashMoE. On this real hardware setup, FlashMoE improves cache hit rate by up to 51% over well-known offloading policies such as LRU and LFU, and achieves up to 2.6x speedup compared to existing MoE inference systems.
Paper Structure (14 sections, 4 equations, 7 figures, 2 tables, 1 algorithm)

This paper contains 14 sections, 4 equations, 7 figures, 2 tables, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND & MOTIVATION
  3. Mixture of Experts
  4. Challenges in Memory Consumption of MoE
  5. Cache System of MoE
  6. FlashMoE SYSTEM
  7. Overall System
  8. ML-Based Cache Replacement Policy
  9. EVALUATION
  10. Evaluation Environment
  11. Initial Model Loading & Prefill Latency
  12. Cache Hit-Rate
  13. Inference latency
  14. Conclusion

Figures (7)

  • Figure 1: Illustration of an MoE model decoder layer.
  • Figure 2: Heatmap comparing LRU and Belady cache policies. The deepest color means the experts are routed at that time step, and the color changes lighter as time goes backward. Evictions based on Belady policy is marked as red 'X' shape, while LRU is marked as green 'O'. Due to LRU's recency-based replacement policy, it fails to evict experts immediately after routing as the optimal algorithm does. Additionally, frequently accessed experts are sometimes evicted just before routing, highlighting a key limitation of LRU. This indicates the need for incorporating frequency-based metrics such as LFU and leveraging strategies.
  • Figure 3: (a) Overall system of FlashMoE. (b) FlashMoE's prefill process.
  • Figure 4: Overall training method illustration of FlashMoE.
  • Figure 5: Decoder layer computation pipeline of FlashMoE.
  • ...and 2 more figures