Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism

Xinglin Pan, Shaohuai Shi, Wenxiang Lin, Yuxin Wang, Zhenheng Tang, Wei Wang, Xiaowen Chu

TL;DR

FinDEP tackles memory- and latency-intensive MoE inference by introducing fine-grained task scheduling for Disaggregated Expert Parallelism. It builds linear performance models for computation and communication, compares attention/shared-expert ordering (ASAS vs AASS), and solves a convex optimization-based search to determine $m_a$, $r_1$, $m_e$, and $r_2$ for maximum throughput, with an offline/online pipeline for real-time workloads. Empirical results across four GPU testbeds and two MoE backbones show FinDEP achieves up to $1.61\times$ speedup over PPPipe and up to $1.24\times$ on a 32-GPU system, while the solver completes in under a second, enabling rapid adaptation to dynamic sequence lengths and batch sizes. The work demonstrates practical impact by reducing idle GPU time, hiding communication overheads, and enabling scalable deployment of large MoE models in serving environments.

Abstract

The mixture-of-experts (MoE) architecture scales model size with sublinear computational increase but suffers from memory-intensive inference due to KV caches and sparse expert activation. Recent disaggregated expert parallelism (DEP) distributes attention and experts to dedicated GPU groups but lacks support for shared experts and efficient task scheduling, limiting performance. We propose FinDEP, a fine-grained task scheduling algorithm for DEP that maximizes task overlap to improve MoE inference throughput. FinDEP introduces three innovations: 1) partitioning computation/communication into smaller tasks for fine-grained pipelining, 2) formulating a scheduling optimization supporting variable granularity and ordering, and 3) developing an efficient solver for this large search space. Experiments on four GPU systems with DeepSeek-V2 and Qwen3-MoE show FinDEP improves throughput by up to 1.61x over prior methods, achieving up to 1.24x speedup on a 32-GPU system.

Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism

TL;DR

FinDEP tackles memory- and latency-intensive MoE inference by introducing fine-grained task scheduling for Disaggregated Expert Parallelism. It builds linear performance models for computation and communication, compares attention/shared-expert ordering (ASAS vs AASS), and solves a convex optimization-based search to determine , , , and for maximum throughput, with an offline/online pipeline for real-time workloads. Empirical results across four GPU testbeds and two MoE backbones show FinDEP achieves up to speedup over PPPipe and up to on a 32-GPU system, while the solver completes in under a second, enabling rapid adaptation to dynamic sequence lengths and batch sizes. The work demonstrates practical impact by reducing idle GPU time, hiding communication overheads, and enabling scalable deployment of large MoE models in serving environments.

Abstract

The mixture-of-experts (MoE) architecture scales model size with sublinear computational increase but suffers from memory-intensive inference due to KV caches and sparse expert activation. Recent disaggregated expert parallelism (DEP) distributes attention and experts to dedicated GPU groups but lacks support for shared experts and efficient task scheduling, limiting performance. We propose FinDEP, a fine-grained task scheduling algorithm for DEP that maximizes task overlap to improve MoE inference throughput. FinDEP introduces three innovations: 1) partitioning computation/communication into smaller tasks for fine-grained pipelining, 2) formulating a scheduling optimization supporting variable granularity and ordering, and 3) developing an efficient solver for this large search space. Experiments on four GPU systems with DeepSeek-V2 and Qwen3-MoE show FinDEP improves throughput by up to 1.61x over prior methods, achieving up to 1.24x speedup on a 32-GPU system.
Paper Structure (20 sections, 4 theorems, 13 equations, 7 figures, 7 tables, 1 algorithm)

This paper contains 20 sections, 4 theorems, 13 equations, 7 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivations
  3. MoE Layer
  4. Disaggregated Expert Parallelism and Ping-pong Pipeline
  5. Motivations
  6. Problem Formulation
  7. Execution Time Formulation
  8. Optimization Problem Formulation
  9. Solution
  10. Performance Models
  11. Determine Task Order, $m_a$, $r_1$, $m_e$, and $r_2$.
  12. Algorithm
  13. Evaluation
  14. Testbeds
  15. Verification of Performance Models
  16. ...and 5 more sections

Key Result

Theorem 1

Given pair $(r_1, r_2)$, the objective function in Eq. eq:simplify-objective is monotonically increasing with respect to $m_a$ .

Figures (7)

  • Figure 1: A typical structure of an MoE model. MLA refers to Multi-Head Latent Attention deepseek_v2, while MHA denotes Multi-Head Attention mha. The "Shared" block indicates one shared expert or several shared experts, which may be optional depending on the MoE configuration.
  • Figure 2: An illustration of DEP. GPUs are partitioned into two groups: AG and EG. AG handles the attention and shared expert computation, while EG handles experts computation.
  • Figure 3: Timeline of naive DEP, PPPipe, and our FinDEP.
  • Figure 4: Comparative examples highlighting the advantages and limitations of AASS and ASAS scheduling strategies.
  • Figure 5: Diagram of the 0-th layer start timestamp $\tau_{e2a}^{(0, i, j)}$, decomposed into three components: pipeline time, fine-grained pipeline time, and initial latency.
  • ...and 2 more figures

Theorems & Definitions (4)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4