ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling

TL;DR

ZipMoE tackles the challenge of deploying large Mixture-of-Experts (MoE) models on resource-constrained edge devices by combining lossless parameter compression with a cache-affinity scheduling strategy tailored to UMA-based hardware. It introduces a bit-field decomposition that separates exponent bits (E-chunks) from sign–mantissa bits (SM-chunks) and leverages hierarchical cache pools and a DAG-based, CPU–GPU reconstruction pipeline to enable CPU-parallel decompression and memory-coalesced tensor recovery. The paper provides a constant-factor approximation guarantee for its scheduling algorithm and demonstrates substantial practical gains, achieving up to 72.77% reduction in latency and up to 6.76x throughput improvements over state-of-the-art systems on edge platforms. By preserving full-precision model behavior and avoiding lossy quantization, ZipMoE offers a viable path for secure, real-time MoE inference on mobile and embedded devices, leveraging shared memory in modern SoCs to maximize compute utilization.

Abstract

While Mixture-of-Experts (MoE) architectures substantially bolster the expressive power of large-language models, their prohibitive memory footprint severely impedes the practical deployment on resource-constrained edge devices, especially when model behavior must be preserved without relying on lossy quantization. In this paper, we present ZipMoE, an efficient and semantically lossless on-device MoE serving system. ZipMoE exploits the synergy between the hardware properties of edge devices and the statistical redundancy inherent to MoE parameters via a caching-scheduling co-design with provable performance guarantee. Fundamentally, our design shifts the paradigm of on-device MoE inference from an I/O-bound bottleneck to a compute-centric workflow that enables efficient parallelization. We implement a prototype of ZipMoE and conduct extensive experiments on representative edge computing platforms using popular open-source MoE models and real-world workloads. Our evaluation reveals that ZipMoE achieves up to $72.77\%$ inference latency reduction and up to $6.76\times$ higher throughput than the state-of-the-art systems.

Figures (10)

• Figure 1: Latency break-down of decoding layers in representitive MoE models on (a) Server environment, where experts are offloaded to CPU with $512$GB RAM; and (b) Edge environment, where experts are offloaded to NVMe SSD (Aigo DP35) with $2$GB/s read speed.
• Figure 2: The probability mass heat maps of integer representations of the exponent bits extracted from different MoE parameters. The Shannon entropy values for the three models are $2.651$ bits, $2.563$, and $2.554$ bits, respectively.
• Figure 3: The compression ratios of different MoE parameters using different lossless compressors.
• Figure 4: Comparison between decompression delay when I/O delay in various cases, where D. denotes the existence of decompression in other threads, P. denotes page cache is enabled. All read operations are tested to read the same amount of bytes with decompressed tensors (the size of exponent bits), while full tensor represents the end-to-end delay of transferring the complete tensor to GPU. All experiments are carried out on Jetson AGX Orin 64G with Samsung 970 EVO SSD.
• Figure 5: ZipMoE System Overview

Theorems & Definitions (16)

• Theorem 3.1
• Theorem 3.2
• Definition 1.1: Compute-Bound
• Definition 2.1: Charge
• Definition 2.2: Critical-Path
• Lemma 2.3
• proof
• Corollary 2.4
• Lemma 2.5
• proof