ZipAR: Parallel Auto-regressive Image Generation through Spatial Locality

TL;DR

ZipAR addresses the slow sampling of autoregressive visual generation by exploiting the spatial locality of visual data to decode multiple tokens across adjacent rows in parallel. It is training-free and plug-and-play, using a local window and an adaptive window assignment scheme to maintain image fidelity while dramatically reducing forward passes. Empirical results across multiple AR models and datasets show substantial speedups (up to 91% fewer forward steps) with minimal quality loss, and the method remains compatible with other acceleration techniques. This approach has practical significance for more efficient, lower-latency, and potentially greener deployment of visual generation models.

Abstract

In this paper, we propose ZipAR, a training-free, plug-and-play parallel decoding framework for accelerating auto-regressive (AR) visual generation. The motivation stems from the observation that images exhibit local structures, and spatially distant regions tend to have minimal interdependence. Given a partially decoded set of visual tokens, in addition to the original next-token prediction scheme in the row dimension, the tokens corresponding to spatially adjacent regions in the column dimension can be decoded in parallel, enabling the ``next-set prediction'' paradigm. By decoding multiple tokens simultaneously in a single forward pass, the number of forward passes required to generate an image is significantly reduced, resulting in a substantial improvement in generation efficiency. Experiments demonstrate that ZipAR can reduce the number of model forward passes by up to 91% on the Emu3-Gen model without requiring any additional retraining. Code is available here: https://github.com/ThisisBillhe/ZipAR.

Figures (7)

• Figure 1: Up to 91% forward step reduction with ZipAR. Samples are generated by Emu3-Gen model with next-token prediction paradigm (the first column) and ZipAR (the right three columns).
• Figure 2: (a) An overview of the training and decoding pipeline for autoregressive (AR) visual generation models. For models trained with a next-token prediction objective, each forward pass generates a single visual token. (b) Medusa cai2024medusa and Jacobi santilli2023jacobi decoding predict multiple adjacent tokens in sequence order. (c) MAR li2024mar predicts multiple tokens in a random order. (d) The proposed ZipAR predicts multiple spatially adjacent tokens.
• Figure 3: The attention scores of visual tokens in the Lumina-mGPT-7B liu2024luminamgpt and LlamaGen-XL sun2024llamagen models. Slash lines indicate that significant attention scores are allocated to tokens at fixed intervals, corresponding to tokens in the same column of previous rows. The full attention scores are presented by storing the attention scores of each visual token during decoding and concatenating them.
• Figure 4: A toy example of the ZipAR framework. The window size is set to $2$ in this toy example.
• Figure 5: The local window size required to retain 95% of attention scores across different rows and input prompt. Data is collected from the first token of each row in Lumina-mGPT-7B model with input prompt from COCO lin2014coco and Parti yu2022parti dataset.