Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeServe: Towards Affordable Offline LLM Inference via Decentralization

Linyu Wu, Xiaoyuan Liu, Tianneng Shi, Zhe Ye, Dawn Song

TL;DR

DeServe tackles the high cost and limited availability of GPU resources for offline LLM inference by enabling a decentralized, offline-serving marketplace that leverages idle GPUs. It combines inter-layer (pipeline) model parallelism with KV cache offloading and microbatch scheduling to mitigate high-latency network bubbles and boost throughput, supported by a formal cost/profit model. The approach achieves substantial throughput gains ($6.7\times$ to $12.6\times$) in high-latency environments and is complemented by a modular on-chain framework for task registration, payments, and arbitration to protect correctness and incentives. This work highlights a practical path toward affordable offline LLM inference at scale, while also outlining open problems in decentralization, verification, and governance.

Abstract

The rapid growth of generative AI and its integration into everyday workflows have significantly increased the demand for large language model (LLM) inference services. While proprietary models remain popular, recent advancements in open-source LLMs have positioned them as strong contenders. However, deploying these models is often constrained by the high costs and limited availability of GPU resources. In response, this paper presents the design of a decentralized offline serving system for LLM inference. Utilizing idle GPU resources, our proposed system, DeServe, decentralizes access to LLMs at a lower cost. DeServe specifically addresses key challenges in optimizing serving throughput in high-latency network environments. Experiments demonstrate that DeServe achieves a 6.7x-12.6x improvement in throughput over existing serving system baselines in such conditions.

DeServe: Towards Affordable Offline LLM Inference via Decentralization

TL;DR

DeServe tackles the high cost and limited availability of GPU resources for offline LLM inference by enabling a decentralized, offline-serving marketplace that leverages idle GPUs. It combines inter-layer (pipeline) model parallelism with KV cache offloading and microbatch scheduling to mitigate high-latency network bubbles and boost throughput, supported by a formal cost/profit model. The approach achieves substantial throughput gains ( to ) in high-latency environments and is complemented by a modular on-chain framework for task registration, payments, and arbitration to protect correctness and incentives. This work highlights a practical path toward affordable offline LLM inference at scale, while also outlining open problems in decentralization, verification, and governance.

Abstract

The rapid growth of generative AI and its integration into everyday workflows have significantly increased the demand for large language model (LLM) inference services. While proprietary models remain popular, recent advancements in open-source LLMs have positioned them as strong contenders. However, deploying these models is often constrained by the high costs and limited availability of GPU resources. In response, this paper presents the design of a decentralized offline serving system for LLM inference. Utilizing idle GPU resources, our proposed system, DeServe, decentralizes access to LLMs at a lower cost. DeServe specifically addresses key challenges in optimizing serving throughput in high-latency network environments. Experiments demonstrate that DeServe achieves a 6.7x-12.6x improvement in throughput over existing serving system baselines in such conditions.
Paper Structure (23 sections, 2 equations, 4 figures, 4 tables)

This paper contains 23 sections, 2 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Transformer Layers and Auto-regressive Decoding
  4. Online and Offline Serving System
  5. Model Parallelism and Partition
  6. Intra-layer Parallelism
  7. Inter-layer Parallelism
  8. Motivation and Cost Model
  9. Cost of Different Compute Resources
  10. Pricing for LLM Services
  11. Profit Model
  12. Optimize Decentralized Serving
  13. Limited memory for each batch.
  14. Network latency bubbles.
  15. Overview
  16. ...and 8 more sections

Figures (4)

  • Figure 1: DeServe Framework Overview. It matches users with LLM inference needs and miners with GPUs, while facilitating payments and inference correctness arbitrations.
  • Figure 2: Pipeline parallelism in centralized and decentralized environment.
  • Figure 3: Memory layout of DeServe with 4 microbatches. Model represents the model weights. L0, L1, ... represent local page pools for each microbatch. G0, G1 represent global page pools.
  • Figure 4: Overlapping KV cache swapping with LLM inference. LLM inference computation is represented in yellow, KV cache swapping in is in green, and KV cache swapping out is in red.