Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speculative Contrastive Decoding

Hongyi Yuan, Keming Lu, Fei Huang, Zheng Yuan, Chang Zhou

TL;DR

Speculative Contrastive Decoding~(SCD), a straightforward yet powerful decoding approach that leverages predictions from smaller language models~(LMs) to achieve both decoding acceleration and quality improvement, is introduced.

Abstract

Large language models~(LLMs) exhibit exceptional performance in language tasks, yet their auto-regressive inference is limited due to high computational requirements and is sub-optimal due to the exposure bias. Inspired by speculative decoding and contrastive decoding, we introduce Speculative Contrastive Decoding~(SCD), a straightforward yet powerful decoding approach that leverages predictions from smaller language models~(LMs) to achieve both decoding acceleration and quality improvement. Extensive evaluations and analyses on four diverse language tasks demonstrate the effectiveness of SCD, showing that decoding efficiency and quality can compatibly benefit from one smaller LM.

Speculative Contrastive Decoding

TL;DR

Speculative Contrastive Decoding~(SCD), a straightforward yet powerful decoding approach that leverages predictions from smaller language models~(LMs) to achieve both decoding acceleration and quality improvement, is introduced.

Abstract

Large language models~(LLMs) exhibit exceptional performance in language tasks, yet their auto-regressive inference is limited due to high computational requirements and is sub-optimal due to the exposure bias. Inspired by speculative decoding and contrastive decoding, we introduce Speculative Contrastive Decoding~(SCD), a straightforward yet powerful decoding approach that leverages predictions from smaller language models~(LMs) to achieve both decoding acceleration and quality improvement. Extensive evaluations and analyses on four diverse language tasks demonstrate the effectiveness of SCD, showing that decoding efficiency and quality can compatibly benefit from one smaller LM.
Paper Structure (16 sections, 2 theorems, 4 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 16 sections, 2 theorems, 4 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Contrastive Decoding
  5. Speculative Decoding
  6. Speculative Contrastive Decoding
  7. Experiment
  8. Analysis
  9. Conclusion
  10. Experiment Details
  11. Benchmark Details
  12. Configuration Details
  13. Hyper-parameter Details
  14. Proof of Theorem \ref{['theo:main']}
  15. Case Study
  16. ...and 1 more sections

Key Result

Theorem 5.1

The expected acceleration factor in decoding runtime is $\frac{1-\lambda^{\gamma+1}}{(1-\lambda)(1 + c\gamma + c\lambda^\gamma)}$.

Figures (5)

  • Figure 1: Hyper-parameter analysis on expected acceleration factors regarding empirical acceptance rate $\lambda$. The best hyper-parameter settings as in \ref{['tab:main_table']} are the lines marked with triangles.
  • Figure 2: The averaged token distribution entropy with error bars of rejected and accepted tokens in SCD.
  • Figure 3: Performance sensitivity regarding $\alpha$ and $\beta$.
  • Figure 4: Hyper-parameter analysis on expected acceleration factors regarding empirical acceptance rate $\lambda$. The best hyper-parameter settings as in \ref{['tab:main_table']} are the lines marked with triangles.
  • Figure 5: Performance sensitivity regarding $\alpha$ and $\beta$.

Theorems & Definitions (3)

  • Theorem 5.1
  • Theorem B.1
  • proof