Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs

Jinwoo Park, Seunggeun Cho, Dongsu Han

TL;DR

SpecEdge tackles the cost and latency barriers of large-scale LLM serving by splitting drafting between edge GPUs and verification on servers, exchanging only token outputs to minimize network traffic. It introduces proactive edge drafting and pipeline-aware server scheduling to hide WAN latency and keep GPUs busy. Across experiments, SpecEdge delivers around 1.9–2.0× cost efficiency and about 2.2× server throughput with notable inter-token latency reductions under WAN conditions. The approach preserves the server-model output distribution and generalizes across models, suggesting a scalable, cost-effective paradigm for interactive LLM serving.

Abstract

Large language models (LLMs) power many modern applications, but serving them at scale remains costly and resource-intensive. Current server-centric systems overlook consumer-grade GPUs at the edge. We introduce SpecEdge, an edge-assisted inference framework that splits LLM workloads between edge and server GPUs using a speculative decoding scheme, exchanging only token outputs over the network. SpecEdge employs proactive edge drafting to overlap edge token creation with server verification and pipeline-aware scheduling that interleaves multiple user requests to increase server-side throughput. Experiments show SpecEdge enhances overall cost efficiency by 1.91x through achieving 2.22x server throughput, and reduces inter token latency by 11.24% compared to a server-only baseline, introducing a scalable, cost-effective paradigm for LLM serving. The code is available at https://github.com/kaist-ina/specedge

SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs

TL;DR

SpecEdge tackles the cost and latency barriers of large-scale LLM serving by splitting drafting between edge GPUs and verification on servers, exchanging only token outputs to minimize network traffic. It introduces proactive edge drafting and pipeline-aware server scheduling to hide WAN latency and keep GPUs busy. Across experiments, SpecEdge delivers around 1.9–2.0× cost efficiency and about 2.2× server throughput with notable inter-token latency reductions under WAN conditions. The approach preserves the server-model output distribution and generalizes across models, suggesting a scalable, cost-effective paradigm for interactive LLM serving.

Abstract

Large language models (LLMs) power many modern applications, but serving them at scale remains costly and resource-intensive. Current server-centric systems overlook consumer-grade GPUs at the edge. We introduce SpecEdge, an edge-assisted inference framework that splits LLM workloads between edge and server GPUs using a speculative decoding scheme, exchanging only token outputs over the network. SpecEdge employs proactive edge drafting to overlap edge token creation with server verification and pipeline-aware scheduling that interleaves multiple user requests to increase server-side throughput. Experiments show SpecEdge enhances overall cost efficiency by 1.91x through achieving 2.22x server throughput, and reduces inter token latency by 11.24% compared to a server-only baseline, introducing a scalable, cost-effective paradigm for LLM serving. The code is available at https://github.com/kaist-ina/specedge

Paper Structure

This paper contains 21 sections, 1 equation, 16 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Problem and Motivation
  4. SpecEdge Design
  5. Disaggregated LLM Decoding
  6. Proactive Draft Generation at the Edge
  7. Server-side Pipeline-aware Scheduling
  8. Evaluation
  9. End-to-end Performance and Cost-efficiency
  10. Component-wise Benefit
  11. System Sensitivity Analysis
  12. Conclusion
  13. Additional Details on Evaluation Setup
  14. Comprehensive Performance Analysis
  15. Performance with Diverse Models and Datasets
  16. ...and 6 more sections

Figures (16)

  • Figure 1: Token generation cost comparison with Qwen2-0.5B model.
  • Figure 2: Abstract timeline of SpecEdge with draft (edge-side) and verify (server-side) inference concept.
  • Figure 3: A speculative decoding cycle composed of draft and verify stages.
  • Figure 4: Illustrative example of a proactive draft tree expansion.
  • Figure 5: Post-verification update with complete draft alignment and subsequent draft submission.
  • ...and 11 more figures