Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement

TL;DR

This paper tackles delivering Mixture-of-Experts (MoE) inference on memory-limited, heterogeneous edge servers. It introduces Prism, a distributed inference framework with a global scheduler and activation-aware placement that minimizes cross-server communication under per-GPU memory constraints, formalized by a proxy objective for remote invocations $\min_{\mathcal{P}} \sum_{n=1}^N \sum_{l=1}^L \sum_{e \in \mathcal{E}_l} f_n^l(e) \cdot \mathbf{1}_{\text{remote}}(n,e)$ and a migration policy using $T_{\text{mig}}$ and $C(\mathcal{P})$. The system employs a two-stage placement: (i) layer-wise expert count allocation guided by activation diversity (entropy) and (ii) expert-to-server assignment with a greedy, coverage-guaranteeing strategy, followed by lightweight migrations when workload shifts justify the cost. Empirical results on Mixtral-8x7B and DeepSeek-V2-Lite across edge hardware show up to 30.6% reduction in inference latency and notable decreases in cross-server communication compared with baselines, validating Prism’s effectiveness for cooperative edge MoE serving. These findings indicate a practical pathway to scalable, privacy-preserving, edge-resident MoE inference in heterogeneous deployments.

Abstract

The emergence of Mixture-of-Experts (MoE) has transformed the scaling of large language models by enabling vast model capacity through sparse activation. Yet, converting these performance gains into practical edge deployment remains difficult, as the massive memory footprint and communication demands often overwhelm resource-limited environments. While centralized cloud-based solutions are available, they are frequently plagued by prohibitive infrastructure costs, latency issues, and privacy concerns. Moreover, existing edge-oriented optimizations largely overlook the complexities of heterogeneous hardware, focusing instead on isolated or uniform device setups. In response, this paper proposes Prism, an inference framework engineered for collaborative MoE serving across diverse GPU-equipped edge servers. By leveraging the intrinsic sparsity and input locality of MoE workloads, Prism minimizes inter-server communication and optimizes expert placement within diverse resource constraints. The framework integrates an activation-aware placement strategy that balances local request coverage with memory utilization, supplemented by a runtime migration mechanism to adapt expert distribution to dynamic workload changes. Experiments on contemporary MoE models and datasets demonstrate that Prism reduces inference latency by up to 30.6% and significantly lowers communication costs compared to state-of-the-art baselines, confirming the effectiveness of cooperative edge-based MoE serving.

Figures (8)

• Figure 1: Illustration of distributed MoE inference across three edge servers. Each server handles requests from its own users and hosts a subset of experts. When a required expert is not available locally, the server performs remote computation by invoking the expert on another node.
• Figure 2: Activation patterns across tasks.
• Figure 3: Activation patterns across layers.
• Figure 4: The workflow of Prism. The system consists of two primary components working in coordination to enable efficient distributed inference for MoE models: a global scheduler and a runtime multi-server system that executes inference.
• Figure 5: Layer-wise inference latency increases with the proportion of experts executed on remote servers.

Theorems & Definitions (4)

• Lemma 1: Entropy-Guided Coverage Lower Bound
• proof : Proof sketch.
• Theorem 1: Greedy Approximation for Local Expert Assignment
• proof : Proof sketch.