Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification

Haoyun Jiang, Junqi He, Feng Hong, Xinlong Yang, Jianwei Zhang, Zheng Li, Zhengyang Zhuge, Zhiyong Chen, Bo Han, Junyang Lin, Jiangchao Yao

TL;DR

TriSpec addresses the verification bottleneck in speculative decoding for large language models by introducing a lightweight proxy verifier that collaborates with a single-layer drafter and the target model. A margin-based routing mechanism decides when to trust proxy verification and when to escalate to the full target, effectively offloading a large fraction of verification work. The approach preserves accuracy within a small margin while delivering substantial throughput gains across multiple model families and benchmarks, with reported speedups up to 30–35% and reduced target invocations. This work demonstrates that verification-time offloading to lightweight proxies is a practical and scalable path to faster, more efficient LLM inference.

Abstract

Inference efficiency in Large Language Models (LLMs) is fundamentally limited by their serial, autoregressive generation, especially as reasoning becomes a key capability and response sequences grow longer. Speculative decoding (SD) offers a powerful solution, providing significant speed-ups through its lightweight drafting and parallel verification mechanism. While existing work has nearly saturated improvements in draft effectiveness and efficiency, this paper advances SD from a new yet critical perspective: the verification cost. We propose TriSpec, a novel ternary SD framework that, at its core, introduces a lightweight proxy to significantly reduce computational cost by approving easily verifiable draft sequences and engaging the full target model only when encountering uncertain tokens. TriSpec can be integrated with state-of-the-art SD methods like EAGLE-3 to further reduce verification costs, achieving greater acceleration. Extensive experiments on the Qwen3 and DeepSeek-R1-Distill-Qwen/LLaMA families show that TriSpec achieves up to 35\% speedup over standard SD, with up to 50\% fewer target model invocations while maintaining comparable accuracy.

TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification

TL;DR

TriSpec addresses the verification bottleneck in speculative decoding for large language models by introducing a lightweight proxy verifier that collaborates with a single-layer drafter and the target model. A margin-based routing mechanism decides when to trust proxy verification and when to escalate to the full target, effectively offloading a large fraction of verification work. The approach preserves accuracy within a small margin while delivering substantial throughput gains across multiple model families and benchmarks, with reported speedups up to 30–35% and reduced target invocations. This work demonstrates that verification-time offloading to lightweight proxies is a practical and scalable path to faster, more efficient LLM inference.

Abstract

Inference efficiency in Large Language Models (LLMs) is fundamentally limited by their serial, autoregressive generation, especially as reasoning becomes a key capability and response sequences grow longer. Speculative decoding (SD) offers a powerful solution, providing significant speed-ups through its lightweight drafting and parallel verification mechanism. While existing work has nearly saturated improvements in draft effectiveness and efficiency, this paper advances SD from a new yet critical perspective: the verification cost. We propose TriSpec, a novel ternary SD framework that, at its core, introduces a lightweight proxy to significantly reduce computational cost by approving easily verifiable draft sequences and engaging the full target model only when encountering uncertain tokens. TriSpec can be integrated with state-of-the-art SD methods like EAGLE-3 to further reduce verification costs, achieving greater acceleration. Extensive experiments on the Qwen3 and DeepSeek-R1-Distill-Qwen/LLaMA families show that TriSpec achieves up to 35\% speedup over standard SD, with up to 50\% fewer target model invocations while maintaining comparable accuracy.
Paper Structure (32 sections, 1 theorem, 10 equations, 10 figures, 8 tables, 1 algorithm)

This paper contains 32 sections, 1 theorem, 10 equations, 10 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Rethinking Speculative Decoding Speedup
  3. Speculative Decoding
  4. Latency Analysis of Speculative Decoding
  5. TriSpec
  6. Proxy choice: same-family smaller models
  7. Ternary speculative decoding
  8. Further Discussion of TriSpec
  9. Experiments
  10. Setup
  11. Effectiveness of TriSpec
  12. Routing Criteria
  13. Ablation Studies
  14. Related Work
  15. Conclusion
  16. ...and 17 more sections

Key Result

Lemma 2.1

Consider a speculative decoding process that generates $N$ tokens. Let $\tau$ denote the average acceptance length (tokens accepted per verification round), and let $t_d$, $t_v$, and $t_o$ be the average per-round drafting time, verification time, and other overhead, respectively. Then the end-to-en In practice, $t_o$ is typically small and nearly constant compared to $t_d$ and $t_v$, and $N$ can

Figures (10)

  • Figure 1: Comparison of standard speculative decoding and TriSpec. Case I: The proxy verifier provides local correction of drafts, bypassing the large-scale target model to gain speed. Case II: If the proxy's verification is deemed untrusted before the first rejection, the corresponding tokens are escalated to the target model for authoritative validation to ensure accuracy. TriSpec employs a margin-based criterion to classify proxy verification as trusted or untrusted.
  • Figure 2: (a) Latency decomposition of speculative decoding across representative methods. EAGLE reduces drafting cost $t_d$ via a single-layer design. EAGLE-3 and related methods further enlarge the acceptance length $\tau$ by improving draft token quality. Our proposed TriSpec introduces a new perspective by reducing the verification cost $t_v$. (b) Distribution of token outcomes versus the proxy’s top-1–top-2 probability margin on ShareGPT. Areas are stacked and globally normalized, so the total across all classes sums to 1.
  • Figure 3: Training pipeline of the draft model and adapter in TriSpec. The green, yellow, and gray modules represent the target, proxy, and drafter components, respectively.
  • Figure 4: Token pruning and attention mask updates in TriSpec.
  • Figure 5: (a) Radar chart illustrating score and speed across all SpecBench subtasks. All experiments are conducted using models from the Qwen3 family. (b) Accuracy–speed trade-offs of different routing strategies. (c) Decoding speed under varying draft lengths for standard speculative decoding and TriSpec. All experiments are conducted on the MATH500 dataset using Qwen3-32B as the target.
  • ...and 5 more figures

Theorems & Definitions (1)

  • Lemma 2.1: Latency decomposition for SD