HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models

TL;DR

This work tackles the challenge of serving thousands of tenant-specific PLMs on a single GPU by introducing HMI, a Hierarchical knowledge management framework. It splits knowledge into general, domain-specific, and task-specific components and materializes domain knowledge via PLOT (precomputed lower-layer representations) and task knowledge via adapters, enabling efficient multi-tenant inference. Inference proceeds in two phases—representation retrieval from PLOT and high-layer transformer computation with adapters—while pipeline optimizations hide CPU/I/O costs and leverage batched adapters for parallelism. Empirical results show HMI can support up to about $10{,}000$ hPLMs on one GPU with negligible accuracy loss and extends to generative PLMs, offering a practical path to scalable cloud hosting for diverse tenant workloads.

Abstract

The significant computational demands of pretrained language models (PLMs), which often require dedicated hardware, present a substantial challenge in serving them efficiently, especially in multi-tenant environments. To address this, we introduce HMI, a Hierarchical knowledge management-based Multi-tenant Inference system, designed to manage tenants with distinct PLMs resource-efficiently. Our approach is three-fold: Firstly, we categorize PLM knowledge into general, domain-specific, and task-specific. Leveraging insights on knowledge acquisition across different model layers, we construct hierarchical PLMs (hPLMs) by extracting and storing knowledge at different levels, significantly reducing GPU memory usage per tenant. Secondly, we establish hierarchical knowledge management for hPLMs generated by various tenants in HMI. We manage domain-specific knowledge with acceptable storage increases by constructing and updating domain-specific knowledge trees based on frequency. We manage task-specific knowledge within limited GPU memory through parameter swapping. Finally, we propose system optimizations to enhance resource utilization and inference throughput. These include fine-grained pipelining via hierarchical knowledge prefetching to overlap CPU and I/O operations with GPU computations, and optimizing parallel implementations with batched matrix multiplications. Our experimental results demonstrate that the proposed HMI can efficiently serve up to 10,000 hPLMs (hBERTs and hGPTs) on a single GPU, with only a negligible compromise in accuracy.

Figures (17)

• Figure 1: An overview of the multi-tenant inference workflow and the associated challenges when adapted for PLMs.
• Figure 2: Model version tree and the separation of the learned hierarchical knowledge of a hPLM instance.
• Figure 3: Impact of model swapping on GPU cache hit rate and inference efficiency slowdown ratio while serving different numbers of tenants.
• Figure 4: Impact of model compression on throughput while serving different numbers of tenants.
• Figure 5: Performance comparison (Acc/F1) of different domain-specific tasks using parameter-efficient fine-tuning via adapter versus fully fine-tuning.