CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory

TL;DR

CoServe addresses memory bottlenecks in Collaboration-of-Experts (CoE) inference on memory-constrained devices by exploiting expert dependencies during scheduling and memory management on heterogeneous CPU-GPU hardware. It introduces dependency-aware request scheduling, dependency-aware expert management, and an offline profiler to tailor memory allocations, achieving throughput improvements of 4.5x to 12x over the state-of-the-art Samba-CoE in real-world circuit-board inspection workloads. By grouping requests, prioritizing eviction of low-probability/dependency-free experts, and profiling device-specific performance, CoServe reduces expensive expert switching and optimizes resource use for edge CoE deployments. The results demonstrate substantial system-level gains and highlight the practical impact of dependency-aware design for scalable, memory-efficient multi-expert inference.

Abstract

Large language models like GPT-4 are resource-intensive, but recent advancements suggest that smaller, specialized experts can outperform the monolithic models on specific tasks. The Collaboration-of-Experts (CoE) approach integrates multiple expert models, improving the accuracy of generated results and offering great potential for precision-critical applications, such as automatic circuit board quality inspection. However, deploying CoE serving systems presents challenges to memory capacity due to the large number of experts required, which can lead to significant performance overhead from frequent expert switching across different memory and storage tiers. We propose CoServe, an efficient CoE model serving system on heterogeneous CPU and GPU with limited memory. CoServe reduces unnecessary expert switching by leveraging expert dependency, a key property of CoE inference. CoServe introduces a dependency-aware request scheduler and dependency-aware expert management for efficient inference. It also introduces an offline profiler to automatically find optimal resource allocation on various processors and devices. In real-world intelligent manufacturing workloads, CoServe achieves 4.5$\times$ to 12$\times$ higher throughput compared to state-of-the-art systems.

Figures (19)

• Figure 1: Proportion of expert switching latency and execution latency on devices with non-uniform memory architecture (NUMA) and uniform memory architecture (UMA). The SSD in the NUMA system is a MICRON MTFDDAK480TDS, with a read bandwidth of 530 MB/s. In the UMA system, the SSD is an APPLE SSD AP0512Z, with a read bandwidth of approximately 3000 MB/s.
• Figure 2: Diagram of the Collaboration of Experts (CoE) model inference process.
• Figure 3: Example of expert switching caused by the first-come, first-served approach.
• Figure 4: Example of expert eviction using the Least Recently Used (LRU) strategy.
• Figure 5: Trends in average inference latency with increasing batch size on NUMA and UMA devices.