Multi-Scale Local Speculative Decoding for Image Generation

TL;DR

This work introduces Multi-Scale Local Speculative Decoding (MuLo-SD), a novel framework that combines multi-resolution drafting with spatially informed verification to accelerate AR image generation and sets a new state-of-the-art in speculative decoding for image synthesis, bridging the gap between efficiency and fidelity.

Abstract

Autoregressive (AR) models have achieved remarkable success in image synthesis, yet their sequential nature imposes significant latency constraints. Speculative Decoding offers a promising avenue for acceleration, but existing approaches are limited by token-level ambiguity and lack of spatial awareness. In this work, we introduce Multi-Scale Local Speculative Decoding (MuLo-SD), a novel framework that combines multi-resolution drafting with spatially informed verification to accelerate AR image generation. Our method leverages a low-resolution drafter paired with learned up-samplers to propose candidate image tokens, which are then verified in parallel by a high-resolution target model. Crucially, we incorporate a local rejection and resampling mechanism, enabling efficient correction of draft errors by focusing on spatial neighborhoods rather than raster-scan resampling after the first rejection. We demonstrate that MuLo-SD achieves substantial speedups - up to $\mathbf{1.7\times}$ - outperforming strong speculative decoding baselines such as EAGLE-2 and LANTERN in terms of acceleration, while maintaining comparable semantic alignment and perceptual quality. These results are validated using GenEval, DPG-Bench, and FID/HPSv2 on the MS-COCO 5k validation split. Extensive ablations highlight the impact of up-sampling design, probability pooling, and local rejection and resampling with neighborhood expansion. Our approach sets a new state-of-the-art in speculative decoding for image synthesis, bridging the gap between efficiency and fidelity.

Figures (13)

• Figure 1: Multi-Scale Speculative Decoding extends speculative decoding by using a draft model working at a lower resolution than the target model, to enable acceleration through a coarse-to-fine approach. During verification, we exploit spatial locality in autoregressive models to resample only a neighborhood of rejected image tokens, improving efficiency without compromising quality.
• Figure 2: Overview of our proposed method Multi-Scale Local Speculative Decoding (MuLo-SD). Blue indicates draft tokens, green accepted tokens, purple rejected tokens, blank placeholder tokens. indicates sequential operations, parallel operations, a drawing discontinuity due to looping.
• Figure 3: Representation of the local expansion rule. Green accepted and purple rejected tokens. (a) $R_t$, the set of rejected indices under the target mode as in \ref{['eq:relaxed-acceptance']}, (b) shows raster-scan rejection as in standard SD, (c) $R_X$, the newly introduced local expansion around rejected tokens $R_t$ with a radius $l=1$ as in \ref{['eq:union-neighborhoods']}.
• Figure 4: Visual comparison of 1024p image generations. Each example shows its speedup over the base Tar model (bottom-left). Outputs from EAGLE-2 are omitted since, as an exact decoding method, they match the base model. See the supplementary material for full comparisons and prompts.
• Figure 5: We ablate different components of our method: (a) the contribution of loss functions in the up- down- samplers training, (b) the role of probability pooling during the verification process, and (c) comparison between standard rater-scan rejection and our proposed local rejection and expansion. MSD shortened version for multiscale speculative decoding.