Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BlockLLM: Multi-tenant Finer-grained Serving for Large Language Models

Bodun Hu, Jiamin Li, Le Xu, Myungjin Lee, Akshay Jajoo, Geon-Woo Kim, Hong Xu, Aditya Akella

TL;DR

This work introduces BlockLLM, a serving system that leverages component sharing among fine-tuned LLM models to provide an efficient and flexible solution for LLM workloads, and reduces memory and storage footprints and improves computational efficiency.

Abstract

The increasing demand for Large Language Models (LLMs) across various applications has led to a significant shift in the design of deep learning serving systems. Deploying LLMs, particularly in multi-tenant environments, poses substantial challenges due to their high computational and memory demands. We introduce BlockLLM, a serving system that leverages component sharing among fine-tuned LLM models to provide an efficient and flexible solution for LLM workloads. BlockLLM partitions models into finer-grained blocks, enabling the reuse of model components and independent provisioning to improve computation efficiency. BlockLLM comprises an offline block zoo for storing blocks and an online system to serve requests through chains of blocks. It offers multi-fold flexibilities: (1) Adaptive assembly of blocks on-the-fly through equivalence evaluation among blocks in the zoo; (2) Per-block batch size configuration and best-effort KV cache coordination at the individual block level; (3) Speculative execution and locality-aware block placement to reduce communication costs from dynamic block resource allocation. Our evaluation shows that BlockLLM reduces memory and storage footprints and improves computational efficiency, outperforming existing serving approach in 95%ile latency and GPU utilization by 33.5% and 20.1%, respectively, with minimal impact on accuracy

BlockLLM: Multi-tenant Finer-grained Serving for Large Language Models

TL;DR

This work introduces BlockLLM, a serving system that leverages component sharing among fine-tuned LLM models to provide an efficient and flexible solution for LLM workloads, and reduces memory and storage footprints and improves computational efficiency.

Abstract

The increasing demand for Large Language Models (LLMs) across various applications has led to a significant shift in the design of deep learning serving systems. Deploying LLMs, particularly in multi-tenant environments, poses substantial challenges due to their high computational and memory demands. We introduce BlockLLM, a serving system that leverages component sharing among fine-tuned LLM models to provide an efficient and flexible solution for LLM workloads. BlockLLM partitions models into finer-grained blocks, enabling the reuse of model components and independent provisioning to improve computation efficiency. BlockLLM comprises an offline block zoo for storing blocks and an online system to serve requests through chains of blocks. It offers multi-fold flexibilities: (1) Adaptive assembly of blocks on-the-fly through equivalence evaluation among blocks in the zoo; (2) Per-block batch size configuration and best-effort KV cache coordination at the individual block level; (3) Speculative execution and locality-aware block placement to reduce communication costs from dynamic block resource allocation. Our evaluation shows that BlockLLM reduces memory and storage footprints and improves computational efficiency, outperforming existing serving approach in 95%ile latency and GPU utilization by 33.5% and 20.1%, respectively, with minimal impact on accuracy
Paper Structure (25 sections, 1 equation, 27 figures, 4 tables)

This paper contains 25 sections, 1 equation, 27 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Background
  4. Key Idea: Finer-grained Serving
  5. Comparison with Existing Work
  6. BlockLLM Design
  7. System Overview
  8. Design Challenges
  9. Block Zoo
  10. Equivalence
  11. Accuracy Implications
  12. Model Partitioning
  13. Stitching Block
  14. Online Serving
  15. KV Cache Coordination
  16. ...and 10 more sections

Figures (27)

  • Figure 1: System architecture. $b_x$ are blocks. 'R' are requests. $SB_3$ is the surrogates of Block $b_3$.
  • Figure 1: Percentage of shared parameters of different PE techniques.
  • Figure 2: Example of blocks for two models fine-tuned from the same foundation. We show one Transformer layer for simplicity.
  • Figure 3: Output similarity between Transformer layer groups of two models.
  • Figure 4: Architectural changes of PE.
  • ...and 22 more figures