SJD-PAC: Accelerating Speculative Jacobi Decoding via Proactive Drafting and Adaptive Continuation

Abstract

Speculative Jacobi Decoding (SJD) offers a draft-model-free approach to accelerate autoregressive text-to-image synthesis. However, the high-entropy nature of visual generation yields low draft-token acceptance rates in complex regions, creating a bottleneck that severely limits overall throughput. To overcome this, we introduce SJD-PAC, an enhanced SJD framework. First, SJD-PAC employs a proactive drafting strategy to improve local acceptance rates in these challenging high-entropy regions. Second, we introduce an adaptive continuation mechanism that sustains sequence validation after an initial rejection, bypassing the need for full resampling. Working in tandem, these optimizations significantly increase the average acceptance length per step, boosting inference speed while strictly preserving the target distribution. Experiments on standard text-to-image benchmarks demonstrate that SJD-PAC achieves a $3.8\times$ speedup with lossless image quality.

Figures (6)

• Figure 1: Analysis of acceptance lengths in SJD sjd. The speedup shown in (b) is derived entirely from additionally accepted tokens (acceptance length $> 1$).
• Figure 2: High sensitivity of AR T2I models to token perturbations. Left: Original generation. Right: Replacing a single token (0.04% of total tokens) during the autoregressive process causes a severe visual artifact (red box).
• Figure 3: A conceptual overview of our proposed components compared to Vanilla SJD, illustrating how each method handles a token rejection (red ✗) at iteration $t$. Vanilla SJD(Top): A rejection at position 2 forces a break, and the entire subsequent sequence is resampled (violet $\mathbf{\circlearrowright}$). w/ PD(Middle): After a rejection, Proactive Drafting builds a diverse $K$-ary tree at the rejection point to increase the probability of acceptance in the subsequent iteration. w/ AC(Bottom): After a rejection, Adaptive Continuation continues the verification loop, preserving subsequent valid tokens (violet ✓) instead of terminating.
• Figure 4: Total Variation distance $d_{\text{TV}}$ for text and image generation with respect to the perturbation offset $j$.
• Figure 5: Qualitative Comparison. Images generated using various prompts from the MS-COCO dataset coco. The outputs from our accelerated method (SJD-PAC) are visually indistinguishable from those of the baseline models, corroborating our lossless guarantee.

Theorems & Definitions (2)

• proof
• proof