Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantize-Sample-and-Verify: LLM Acceleration via Adaptive Edge-Cloud Speculative Decoding

Guangyi Zhang, Yunlong Cai, Guanding Yu, Petar Popovski, Osvaldo Simeone

TL;DR

This work tackles bandwidth-limited edge-cloud speculative decoding by preserving the cloud LLM output distribution while enabling on-device drafting. It introduces Quantize-Sample (Q-S), where draft tokens are sampled from a quantized distribution $ ilde{oldsymbol{Q}}^t$, ensuring $ ext{Pr}(X = x_l^t) = p_{l,x_l^t}^t$, i.e., equality with the cloud LLM outputs. A lattice-based quantization scheme is used to discretize probability vectors with controllable bits $b^t$, and a dynamic policy using a double DQN adapts the draft length $L^t$ and quantization precision $b^t$ in response to semantic uncertainty and time-varying channel conditions. The latency-aware framework, validated on CNN/DailyMail summarization with OPT-125M at the edge and OPT-13B in the cloud, demonstrates superior token throughput over static policies and S-Q baselines, with the dynamic approach approaching cloud-only SD performance as bandwidth improves.

Abstract

In edge-cloud speculative decoding (SD), edge devices equipped with small language models (SLMs) generate draft tokens that are verified by large language models (LLMs) in the cloud. A key bottleneck in such systems is the limited communication bandwidth between edge and cloud, which necessitates quantization of the information transmitted about generated tokens. In this work, we introduce a novel quantize-sample (Q-S) strategy that provably preserves the output distribution of the cloud-based model, ensuring that the verified tokens match the distribution of those that would have been generated directly by the LLM. We develop a throughput model for edge-cloud SD that explicitly accounts for communication latency. Leveraging this model, we propose an adaptive mechanism that optimizes token throughput by dynamically adjusting the draft length and quantization precision in response to both semantic uncertainty and channel conditions. Simulations demonstrate that the proposed Q-S approach significantly improves decoding efficiency in realistic edge-cloud deployment scenarios.

Quantize-Sample-and-Verify: LLM Acceleration via Adaptive Edge-Cloud Speculative Decoding

TL;DR

This work tackles bandwidth-limited edge-cloud speculative decoding by preserving the cloud LLM output distribution while enabling on-device drafting. It introduces Quantize-Sample (Q-S), where draft tokens are sampled from a quantized distribution , ensuring , i.e., equality with the cloud LLM outputs. A lattice-based quantization scheme is used to discretize probability vectors with controllable bits , and a dynamic policy using a double DQN adapts the draft length and quantization precision in response to semantic uncertainty and time-varying channel conditions. The latency-aware framework, validated on CNN/DailyMail summarization with OPT-125M at the edge and OPT-13B in the cloud, demonstrates superior token throughput over static policies and S-Q baselines, with the dynamic approach approaching cloud-only SD performance as bandwidth improves.

Abstract

In edge-cloud speculative decoding (SD), edge devices equipped with small language models (SLMs) generate draft tokens that are verified by large language models (LLMs) in the cloud. A key bottleneck in such systems is the limited communication bandwidth between edge and cloud, which necessitates quantization of the information transmitted about generated tokens. In this work, we introduce a novel quantize-sample (Q-S) strategy that provably preserves the output distribution of the cloud-based model, ensuring that the verified tokens match the distribution of those that would have been generated directly by the LLM. We develop a throughput model for edge-cloud SD that explicitly accounts for communication latency. Leveraging this model, we propose an adaptive mechanism that optimizes token throughput by dynamically adjusting the draft length and quantization precision in response to both semantic uncertainty and channel conditions. Simulations demonstrate that the proposed Q-S approach significantly improves decoding efficiency in realistic edge-cloud deployment scenarios.

Paper Structure

This paper contains 17 sections, 1 theorem, 14 equations, 4 figures.

Table of Contents

  1. Introduction
  2. System Model
  3. Token Generation
  4. Uplink Transmission
  5. Token Verification
  6. Downlink Transmission
  7. Quantize-Sample Speculative Decoding
  8. Quantize-Sample-and-Verify
  9. Lattice-Based Quantization
  10. Dynamic Throughput Optimization
  11. Latency Analysis
  12. Problem Definition
  13. Reinforcement Learning-Based Throughput Optimization
  14. Simulation Setup and Evaluation
  15. Setting
  16. ...and 2 more sections

Key Result

Proposition 1

Edge-cloud SD via Q-S guarantees that the probability of generating token $x_l^t$ at iteration $t$, $\mathbb{P}(X = x_l^t)$, coincides with the corresponding probability $p^t_{l,x_l^t}$ for the LLM, i.e.,

Figures (4)

  • Figure 1: Schematic illustration of the proposed quantize-sample (Q-S) scheme for a cloud-edge speculative decoding (SD) system operating over a wireless network. The system operates in each iteration $t$ via the following steps: ① token generation, ② uplink transmission, ③ token verification, and ④ downlink transmission. ($L^t=4$, with superscript $t$ omitted for clarity.)
  • Figure 2: Timeline of the edge–cloud SD operation under the Q-S strategy.
  • Figure 3: Performance versus sampling temperature for both SLM (OPT-125M) and LLM (OPT-13B): (a) ROUGE-2 score; (b) Per-token Shannon entropy.
  • Figure 4: Token throughput achieved by different strategies as a function of sampling temperature. (a) Low-rate regime; (b) High-rate regime.

Theorems & Definitions (1)

  • Proposition 1