HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference

TL;DR

MoE-based LLMs demand substantial memory, constraining edge deployment. HOBBIT introduces a tripartite strategy—token-level dynamic loading with low-precision replacements, layer-level adaptive prefetching, and sequence-level multidimensional caching—to reduce expert-loading latency while preserving accuracy. The approach achieves large decoding speedups (up to 9.93x vs state-of-the-art) and substantial prefill-latency reductions across edge devices, with accuracy loss kept under 1%. Integrated into Llama.cpp, HOBBIT provides a practical, flexible solution for deploying sparse MoE LLMs on memory-constrained platforms.

Abstract

The Mixture-of-Experts (MoE) architecture has demonstrated significant advantages in the era of Large Language Models (LLMs), offering enhanced capabilities with reduced inference costs. However, deploying MoE-based LLMs on memoryconstrained edge devices remains challenging due to their substantial memory requirements. While existing expertoffloading methods alleviate the memory requirements, they often incur significant expert-loading costs or compromise model accuracy. We present HOBBIT, a mixed precision expert offloading system to enable flexible and efficient MoE inference. Our key insight is that dynamically replacing less critical cache-miss experts with low precision versions can substantially reduce expert-loading latency while preserving model accuracy. HOBBIT introduces three innovative techniques that map the natural hierarchy of MoE computation: (1) a token-level dynamic expert loading mechanism, (2) a layer-level adaptive expert prefetching technique, and (3) a sequence-level multidimensional expert caching policy. These innovations fully leverage the benefits of mixedprecision expert inference. By implementing HOBBIT on top of the renowned LLM inference framework Llama.cpp, we evaluate its performance across different edge devices with representative MoE models. The results demonstrate that HOBBIT achieves up to a 9.93x speedup in decoding compared to state-of-the-art MoE offloading systems.

Figures (18)

• Figure 1: Comparison between different LLM architectures.
• Figure 2: Expert-offloading on hardware architecture and model parameter distribution for Mixtral-8x7B.
• Figure 3: Analysis of expert loading acceleration chances.
• Figure 4: System overview of HOBBIT.
• Figure 5: Gating output statistics of Mixtral-8x7B.