Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption

Yifan Tan, Cheng Tan, Zeyu Mi, Haibo Chen

TL;DR

PipeLLM, a user-transparent runtime system that proposes speculative pipelined encryption to predict the data requiring encryption by analyzing the serving patterns of LLMs, and develops an efficient, low-cost pipeline relinquishing approach for instances of incorrect predictions.

Abstract

Confidential computing on GPUs, like NVIDIA H100, mitigates the security risks of outsourced Large Language Models (LLMs) by implementing strong isolation and data encryption. Nonetheless, this encryption incurs a significant performance overhead, reaching up to 52.8 percent and 88.2 percent throughput drop when serving OPT-30B and OPT-66B, respectively. To address this challenge, we introduce PipeLLM, a user-transparent runtime system. PipeLLM removes the overhead by overlapping the encryption and GPU computation through pipelining - an idea inspired by the CPU instruction pipelining - thereby effectively concealing the latency increase caused by encryption. The primary technical challenge is that, unlike CPUs, the encryption module lacks prior knowledge of the specific data needing encryption until it is requested by the GPUs. To this end, we propose speculative pipelined encryption to predict the data requiring encryption by analyzing the serving patterns of LLMs. Further, we have developed an efficient, low-cost pipeline relinquishing approach for instances of incorrect predictions. Our experiments on NVIDIA H100 GPU show that compared with vanilla systems without confidential computing (e.g., vLLM, PEFT, and FlexGen), PipeLLM incurs modest overhead (less than 19.6 percent in throughput) across various LLM sizes, from 13B to 175B.

PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption

TL;DR

PipeLLM, a user-transparent runtime system that proposes speculative pipelined encryption to predict the data requiring encryption by analyzing the serving patterns of LLMs, and develops an efficient, low-cost pipeline relinquishing approach for instances of incorrect predictions.

Abstract

Confidential computing on GPUs, like NVIDIA H100, mitigates the security risks of outsourced Large Language Models (LLMs) by implementing strong isolation and data encryption. Nonetheless, this encryption incurs a significant performance overhead, reaching up to 52.8 percent and 88.2 percent throughput drop when serving OPT-30B and OPT-66B, respectively. To address this challenge, we introduce PipeLLM, a user-transparent runtime system. PipeLLM removes the overhead by overlapping the encryption and GPU computation through pipelining - an idea inspired by the CPU instruction pipelining - thereby effectively concealing the latency increase caused by encryption. The primary technical challenge is that, unlike CPUs, the encryption module lacks prior knowledge of the specific data needing encryption until it is requested by the GPUs. To this end, we propose speculative pipelined encryption to predict the data requiring encryption by analyzing the serving patterns of LLMs. Further, we have developed an efficient, low-cost pipeline relinquishing approach for instances of incorrect predictions. Our experiments on NVIDIA H100 GPU show that compared with vanilla systems without confidential computing (e.g., vLLM, PEFT, and FlexGen), PipeLLM incurs modest overhead (less than 19.6 percent in throughput) across various LLM sizes, from 13B to 175B.

Paper Structure

This paper contains 26 sections, 1 equation, 11 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Large Language Models (LLMs)
  4. Confidential Computing (CC)
  5. Analysis of LLMs on GPU Enclave
  6. Speculative Pipelined Encryption
  7. Main idea
  8. Problem Statement
  9. Speculative Pipelined Encryption
  10. PipeLLM Design
  11. PipeLLM Predictor
  12. Validator: Validating the Consistency of the Ciphertext
  13. Error Handler: Tolerating Wrong IVs with Re-Ordering and NOP Padding
  14. Asynchronous Decryption
  15. Implementation
  16. ...and 11 more sections

Figures (11)

  • Figure 1: Workflow of encrypted data transfer in NVIDIA CC. The messages labeled "a" and "b" represent two consecutive ciphertexts transferred from the CPU to the GPU, while "c" and "d" denote ciphertexts moved from the GPU back to the CPU. After the transfers, the current IV of CPU and GPU is 3 and 7, respectively.
  • Figure 2: Host To Device Memory Copy of Different Data Size
  • Figure 3: confidential computing overhead study.
  • Figure 4: PipeLLM's architecture. Yellow parts are PipeLLM.
  • Figure 5: Common patterns in LLM memory swapping.
  • ...and 6 more figures