Parallelized Autoregressive Visual Generation

TL;DR

The paper tackles slow inference in autoregressive visual generation by introducing PAR, a non-local parallel generation framework. PAR identifies token dependencies and groups weakly dependent distant tokens for parallel prediction while keeping sequential modeling for strongly dependent local tokens, without architectural changes. It integrates into standard autoregressive transformers with a simple reordering and a small set of learnable transition tokens, preserving global context. Experiments on ImageNet and UCF-101 show substantial speedups (up to 9.5x) with comparable or minimally degraded quality across images and videos.

Abstract

Autoregressive models have emerged as a powerful approach for visual generation but suffer from slow inference speed due to their sequential token-by-token prediction process. In this paper, we propose a simple yet effective approach for parallelized autoregressive visual generation that improves generation efficiency while preserving the advantages of autoregressive modeling. Our key insight is that parallel generation depends on visual token dependencies-tokens with weak dependencies can be generated in parallel, while strongly dependent adjacent tokens are difficult to generate together, as their independent sampling may lead to inconsistencies. Based on this observation, we develop a parallel generation strategy that generates distant tokens with weak dependencies in parallel while maintaining sequential generation for strongly dependent local tokens. Our approach can be seamlessly integrated into standard autoregressive models without modifying the architecture or tokenizer. Experiments on ImageNet and UCF-101 demonstrate that our method achieves a 3.6x speedup with comparable quality and up to 9.5x speedup with minimal quality degradation across both image and video generation tasks. We hope this work will inspire future research in efficient visual generation and unified autoregressive modeling. Project page: https://yuqingwang1029.github.io/PAR-project.

Figures (12)

• Figure 1: Comparison of different parallel generation strategies. Both strategies generate initial tokens [1,2,3,4] sequentially then generate multiple tokens in parallel per step, following the order [5a-5d] to [6a-6d] to [7a-7d], etc. (a) Our approach generates weakly dependent tokens across non-local regions in parallel, preserving coherent patterns and local details. (b) The naive method generates strongly dependent tokens within local regions simultaneously, while independent sampling for strongly correlated tokens can cause inconsistent generation and disrupted patterns, such as distorted tiger faces and fragmented zebra stripes.
• Figure 2: Visualization comparison of our parallel generation and traditional autoregressive generation (LlamaGen sun2024autoregressive). Our approach (PAR) achieves 3.6-9.5$\times$ speedup over LlamaGen with comparable quality, reducing the generation time from 12.41s to 3.46s (PAR-4$\times$) and 1.31s (PAR-16$\times$) per image. Time measurements are conducted with a batch size of 1 on a single A100 GPU.
• Figure 3: Illustration of our non-local parallel generation process. Stage 1: sequential generation of initial tokens (1-4) for each region (separated by dotted lines) to establish global structure. Stage 2: parallel generation at aligned positions across different regions (e.g., 5a-5d), then moving to next aligned positions (6a-6d, 7a-7d, etc.) for parallel generation. Same numbers indicate tokens generated in the same step, and letter suffix (a,b,c,d) denotes different regions .
• Figure 4: Overview of our parallel autoregressive generation framework.(a) Model implementation. The model first generates initial tokens sequentially [1,2,3,4], then uses learnable tokens [M1,M2,M3] to help transition into parallel prediction mode. (b) Comparison of visible context between our parallel prediction approach (left) and traditional single-token prediction (right). The colored cells indicate available context during generation. In traditional AR, when predicting token $6d$, the model can access all previous tokens including $6a-6c$. Without full attention, our parallel approach would limit each token (e.g., $6b$) to only see tokens up to the same position in the previous group (e.g., up to $5b$). We enable group-wise full attention to allow access to the entire previous group.
• Figure 5: Qualitative comparison of parallel generation strategies.Top: Our method with sequential initial tokens followed by parallel distant token prediction produces high-quality and coherent images. Middle: Direct parallel prediction without sequential initial tokens leads to inconsistent global structures. Bottom: Parallel prediction of adjacent tokens results in distorted local patterns and broken details.