QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration

TL;DR

This work tackles the memory and throughput challenges of serving multiple fine-tuned mixture-of-experts LLMs on a single-GPU in multi-tenant setups. It introduces a two-pronged approach: similarity-based consolidation of experts to share similar components across models, and runtime partial reconfiguration to swap non-expert layers on demand, thus maintaining high throughput with only a small TTFT overhead. Evaluations on a single NVIDIA A100 (80 GB) using Mixtral-8x7B and Google Switch Transformer Base-8 variants demonstrate an ~85% reduction in turnaround time compared to MIG, with throughput comparable to single-model serving and strong output quality across MT-Bench, MMLU, HellaSwag, and TruthfulQA benchmarks. The approach effectively scales to multiple model variants while preserving QoS and reliability, offering a practical path for multi-tenant MoE deployment on constrained GPU resources.

Abstract

The deployment of mixture-of-experts (MoE) large language models (LLMs) presents significant challenges due to their high memory demands. These challenges become even more pronounced in multi-tenant environments, where shared resources must accommodate multiple models, limiting the effectiveness of conventional virtualization techniques. This paper addresses the problem of efficiently serving multiple fine-tuned MoE-LLMs on a single-GPU. We propose a serving system that employs \textit{similarity-based expert consolidation} to reduce the overall memory footprint by sharing similar experts across models. To ensure output quality, we introduce \textit{runtime partial reconfiguration}, dynamically replacing non-expert layers when processing requests from different models. As a result, our approach achieves a competitive output quality while maintaining throughput comparable to serving a single model while incurring a negligible increase in time-to-first-token (TTFT). Experiments on a server with a single NVIDIA A100 GPU (80GB) using Mixtral-8x7B models demonstrate an 85\% average reduction in turnaround time compared to NVIDIA's multi-instance GPU (MIG). Furthermore, experiments on Google's Switch Transformer Base-8 model with up to four variants demonstrate the scalability and resilience of our approach in maintaining output quality compared to other model merging baselines, highlighting its effectiveness.

Figures (5)

• Figure 1: Diagram of an inference system with a single NVIDIA A100 GPU.
• Figure 2: Expert-to-expert distance between Mixtral base and instruct model: Despite being fine-tuned for different tasks, experts in the same positions exhibit similarity.
• Figure 3: Inference process for each layer of the consolidated model: In the case of an expert hit, the inference is performed using the already loaded expert, which may not necessarily belong to the requested model. However, in the case of a miss, the corresponding expert is loaded from the host's memory to preserve the quality of the generated output.
• Figure 4: The throughput of each serving approach is measured in completed requests per minute. Since NVIDIA MIG utilizes two independent processes to handle requests for different models, the arrival rate for each instance is half of the value shown on the x-axis, and the reported throughput is the sum of the throughput for both instances. The reported values represent the average over five independent runs.
• Figure 5: ROUGE-1 scores (higher is better) on the SAMSum dataset for individual models and their combinations, comparing the proposed approach with the model-merging baseline. Model layouts are illustrated beneath each bar. In our approach, a portion of the expert parameters (squares) and all non-expert parameters (rectangles) are retained from the model specifically trained for the benchmark (Model A, shown in blue). In contrast, the averaging approach modifies all parameters by averaging across the merged models.