Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding

TL;DR

This work tackles the latency of autoregressive text-to-image generation by introducing Speculative Jacobi Decoding (SJD), a training-free probabilistic parallel decoding method that supports sampling-based token decoding. By applying a probabilistic convergence criterion within a sliding Jacobi window and employing spatial-prior token initialization, SJD enables multi-token predictions per iteration and achieves substantial inference speedups without sacrificing visual quality. The approach is validated on multiple AR T2I models (e.g., Lumina-mGPT and Anole) across high-resolution outputs, showing roughly 2× acceleration and up to 2.43× in certain simple-pattern scenarios, with consistent CLIP/FID results. The work preserves the end-to-end training-free paradigm, broadening practical deployment of fast, diverse image generation while maintaining model scalability and quality.

Abstract

The current large auto-regressive models can generate high-quality, high-resolution images, but these models require hundreds or even thousands of steps of next-token prediction during inference, resulting in substantial time consumption. In existing studies, Jacobi decoding, an iterative parallel decoding algorithm, has been used to accelerate the auto-regressive generation and can be executed without training. However, the Jacobi decoding relies on a deterministic criterion to determine the convergence of iterations. Thus, it works for greedy decoding but is incompatible with sampling-based decoding which is crucial for visual quality and diversity in the current auto-regressive text-to-image generation. In this paper, we propose a training-free probabilistic parallel decoding algorithm, Speculative Jacobi Decoding (SJD), to accelerate auto-regressive text-to-image generation. By introducing a probabilistic convergence criterion, our SJD accelerates the inference of auto-regressive text-to-image generation while maintaining the randomness in sampling-based token decoding and allowing the model to generate diverse images. Specifically, SJD facilitates the model to predict multiple tokens at each step and accepts tokens based on the probabilistic criterion, enabling the model to generate images with fewer steps than the conventional next-token-prediction paradigm. We also investigate the token initialization strategies that leverage the spatial locality of visual data to further improve the acceleration ratio under specific scenarios. We conduct experiments for our proposed SJD on multiple auto-regressive text-to-image generation models, showing the effectiveness of model acceleration without sacrificing the visual quality. The code of our work is available here: https://github.com/tyshiwo1/Accelerating-T2I-AR-with-SJD/.

Figures (18)

• Figure 1: We propose Speculative Jacobi Decoding, a training-free multi-token prediction algorithm, to accelerate auto-regressive text-to-image generation by reducing the number of model forward passes (denoted as steps) during inference. We perform our algorithm on Lumina-mGPT, and the reduced steps are marked in red. The original steps are marked in black.
• Figure 2: The results of the greedy decoding (no randomness), top-$10$, top-$100$, and top-$2000$ sampling (high randomness) of Lumina-mGPT liu2024lumina-mgpt. Each row presents the images generated with the same random seeds.
• Figure 3: The pipeline of the vanilla Jacobi decoding on an auto-regressive model. The prediction with sampling is performed in parallel at each Jacobi iteration. We use different shades of blue to indicate the differences between the tokens that have not been accepted.
• Figure 4: Overview of one iteration of our speculative Jacobi decoding (SJD). First, a sequence of draft tokens and the corresponding probabilities are taken as the inputs. Second, we perform a forward pass with the auto-regressive model on the draft tokens, obtaining the probabilities of these tokens. Third, we perform the verification according to these two types of probabilities, accepting a subset of tokens and (re-)sampling the remaining tokens. Last, the accepted tokens are appended to the pre-filling tokens and fixed, while the resampled tokens, along with newly initialized tokens, will serve as the draft tokens for the next iteration.
• Figure 5: The images generated by Lumina-mGPT with different acceleration methods.