Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs

TL;DR

This work proposes Dynamic Expert Sharing (DES), a novel technique that shifts MoE optimization from token-centric pruning and conventional expert skipping methods to sequence-level coreset selection, and introduces two innovative selection strategies: Intra-Sequence Sharing (DES-Seq), which adapts optimal allocation to the sequence level, and Saliency-Aware Voting (DES-Vote), a novel mechanism that allows tokens to collectively elect a coreset based on aggregated router weights.

Abstract

Among parallel decoding paradigms, diffusion large language models (dLLMs) have emerged as a promising candidate that balances generation quality and throughput. However, their integration with Mixture-of-Experts (MoE) architectures is constrained by an expert explosion: as the number of tokens generated in parallel increases, the number of distinct experts activated grows nearly linearly. This results in substantial memory traffic that pushes inference into a memory-bound regime, negating the efficiency gains of both MoE and parallel decoding. To address this challenge, we propose Dynamic Expert Sharing (DES), a novel technique that shifts MoE optimization from token-centric pruning and conventional expert skipping methods to sequence-level coreset selection. To maximize expert reuse, DES identifies a compact, high-utility set of experts to satisfy the requirements of an entire parallel decoding block. We introduce two innovative selection strategies: (1) Intra-Sequence Sharing (DES-Seq), which adapts optimal allocation to the sequence level, and (2) Saliency-Aware Voting (DES-Vote), a novel mechanism that allows tokens to collectively elect a coreset based on aggregated router weights. Extensive experiments on MoE dLLMs demonstrate that DES reduces unique expert activations by over 55% and latency by up to 38%, while retaining 99% of vanilla accuracy, effectively decoupling memory overhead from the degree of parallelism.

Figures (9)

• Figure 1: From left to right: (a) Growth of unique activated experts relative to the degree of parallelism (input block size). (b) Roofline model demonstrating that MoE dLLMs are more memory-bound than their dense counterparts and how DES mitigates this bottleneck. (c) Trade-off between activated experts and relative accuracy, where DES-Vote notably extends the Pareto frontier. (d) MoE kernel latency versus relative accuracy. For (c) and (d), accuracy is reported relative to the vanilla baseline. Results in (c) are averaged across benchmarks, namely HumanEval, MBPP, MATH500, and GSM8K, while (d) evaluates DES-Vote's performance on MBPP. The average activated experts is defined as the mean number of active experts across all MoE layers in the respective model.
• Figure 2: Latency characterization of MoE decoding under varying block lengths. Left: Latency breakdown across MoE FFN, attention, and other components, showing that MoE FFN dominates end-to-end latency. Right: MoE kernel latency versus activated experts for different block lengths (16, 32, 64), illustrating a linear increase in latency with more activated experts.
• Figure 3: Overview of the Dynamic Expert Sharing method. (a) Vanilla MoE dLLM independently routes multiple tokens, leading to a high count of unique activated experts. (b) Dynamic Expert Skipping reduces local per-token computation (indicated by dotted lines) but often fails to optimize the global unique expert load. (c) Dynamic Expert Sharing employs sequence-level coreset selection (via DES-Seq or DES-Vote) to identify high-utility experts globally, significantly minimizing the unique expert weights transferred from HBM. (d) DES-Vote enforces sequence-level consensus by aggregating router weights across the parallel block, contrasting with the independent per-token selection utilized in DES-Seq. Greyed out boxes and bars represent experts not selected and not fetched to on-chip memory.
• Figure 4: Analysis of Dynamic Expert Sharing (DES) in LLaDA-MoE-7B (8 tokens, Layer 10). (Left) Expert Importance: Heatmap of reconstruction loss sensitivity to expert ablation. Darker regions indicate an increase in reconstruction loss when an expert is removed, revealing strong dependencies between specific token positions and experts. (Middle) Selection Overlays: DES-Seq (orange), DES-Vote (pink), and shared (yellow) selections overlaid on log-routing weights. DES-Vote captures high-importance experts (see index 42) that local ranking misses. (Right) Performance vs. Coreset Size ($M_{\text{core}}$): DES-Vote achieves higher Top-$k$ recall (top) and lower residual reconstruction loss (bottom) than DES-Seq across $M_{\text{core}}$. Red dashed lines denote the specific $M_{\text{core}}$ visualized in the heatmaps.
• Figure 5: Latency measurements for MoE kernel (left) and total end-to-end GPU kernel execution time (right) across models.