Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

TL;DR

The paper tackles efficient inference for large Mixture-of-Experts (MoE) models under limited GPU memory by exploiting a heterogeneous CPU-GPU architecture. It introduces Fiddler, a latency-aware system that partitions non-expert and some expert parameters on the GPU and places the rest on CPU memory, then dynamically selects execution strategies for each input token based on per-expert activity and device characteristics. Key contributions include a latency model that captures CPU and GPU batching effects, offline expert-popularity profiling to maximize GPU availability, and a CPU-optimized AVX512_BF16 kernel, all enabling balanced performance across single-batch, long prefill, and beam-search workloads. Empirically, Fiddler outperforms state-of-the-art baselines across these scenarios on Mixtral-8x7B in two hardware setups, demonstrating robust applicability to resource-constrained MoE inference.

Abstract

Large Language Models (LLMs) with the Mixture-of-Experts (MoE) architectures have shown promising performance on various tasks. However, due to the huge model sizes, running them in resource-constrained environments where the GPU memory is not abundant is challenging. Some existing systems propose to use CPU resources to solve that, but they either suffer from the significant overhead of frequently moving data between CPU and GPU, or fail to consider distinct characteristics of CPUs and GPUs. This paper proposes Fiddler, a resource-efficient inference system for MoE models with limited GPU resources. Fiddler strategically utilizes CPU and GPU resources by determining the optimal execution strategy. Our evaluation shows that, unlike state-of-the-art systems that optimize for specific scenarios such as single batch inference or long prefill, Fiddler performs better in all scenarios. Compared against different baselines, Fiddler achieves 1.26 times speed up in single batch inference, 1.30 times in long prefill processing, and 11.57 times in beam search inference. The code of Fiddler is publicly available at https://github.com/efeslab/fiddler.

Figures (12)

• Figure 1: High level overview of Fiddler. Each layer of the MoE model is placed on either the CPU memory or the GPU memory, and Fiddler determines the optimal execution strategy using both the CPU and the GPU based on the number of input tokens of each expert.
• Figure 2: Overview of Fiddler. (a) During the initialization phase, the parameters of non-expert layers and a selected subset of expert layers are allocated to the GPU memory as availability permits; the remaining parameters are allocated to the CPU memory. (b) At runtime, Fiddler dynamically determines the optimal execution strategy by considering the volume of inputs that activate each expert layer along with the different expected latencies of the CPU and the GPU processing.
• Figure 3: Three different scenarios for the execution of expert layers. When the expert weight is present in GPU memory, the expert layer can be executed at GPU without any data transfer (a). When the expert weight is missing in GPU memory, the expert weight can be copied from CPU memory to GPU memory and executed at GPU (b), or the activation can be copied from GPU memory to CPU memory and executed at CPU (c).
• Figure 4: The end-to-end performance comparison by the number of tokens generated per second (scenario , higher is better), with 15 different input/output length configurations. The rightmost set of bars shows the average of 15 configurations.
• Figure 5: The performance comparison by TTFT (scenario , lower is better), with 4 different input length configurations. The rightmost set of bars shows the average of 4 different lengths.