Principled Coarse-Grained Acceptance for Speculative Decoding in Speech

TL;DR

The paper tackles slow autoregressive speech generation caused by strict token-level acceptance in speculative decoding. It introduces Principled Coarse-Graining (PCG), which constructs Acoustic Similarity Groups (ASGs) in the target model's embedding space and performs group-level acceptance with overlap-aware probability splitting, guaranteeing exact sampling at the ASG level. The method uses $P_c$ and $Q_c$ distributions, weight splits $w_{k,t}$, and thinning to sample from the residual, enabling a drop-in AR TTS implementation. On LibriTTS, PCG yields higher acceptance and throughput than standard SD and prior speech-relaxation baselines while preserving WER/CER, Sim-O, and NMOS, demonstrating that acoustically aware group-level acceptance can effectively accelerate speech token generation without compromising quality.

Abstract

Speculative decoding accelerates autoregressive speech generation by letting a fast draft model propose tokens that a larger target model verifies. However, for speech LLMs that generate acoustic tokens, exact token matching is overly restrictive: many discrete tokens are acoustically or semantically interchangeable, reducing acceptance rates and limiting speedups. We introduce Principled Coarse-Graining (PCG), which verifies proposals at the level of Acoustic Similarity Groups (ASGs) derived from the target model's embedding space. By splitting each token's probability mass across the overlapping groups that contain it, we define an overlap-aware coarse-grained distribution and perform rejection sampling on the resulting group variable. This yields an exactness guarantee at the group level while allowing the accepted draft token to stand in for any member of the group in practice. On LibriTTS, PCG increases acceptance and throughput relative to standard speculative decoding and prior speech-specific relaxations while maintaining intelligibility and speaker similarity. These results suggest acoustically aware, group-level acceptance as a simple and general way to accelerate speech token generation while maintaining speech quality.

Figures (4)

• Figure 1: Token-level probabilities used for the SD acceptance criterion: original vs PCG. Colors denote Acoustic Similarity Groups (ASGs); tokens may belong to multiple groups.
• Figure 2: Overview of speculative decoding with PCG. Tokens are first clustered into overlapping ASGs in the target model’s embedding space. The lightweight draft model then autoregressively samples these groups. Finally, the target model verifies all proposed groups in parallel.
• Figure 3: Accuracy–Speedup Trade-off Curve on LibriTTS. While the SSD baseline leads to a sharp drop in accuracy, PCG achieve a more favorable balance between speedup and accuracy. SD denotes standard speculative-decoding.
• Figure 4: Ablation study on the effect of the number of speculated tokens on the speedup of SD with PCG.

Theorems & Definitions (2)

• Proposition
• proof