From Sequential to Spatial: Reordering Autoregression for Efficient Visual Generation

TL;DR

The paper addresses the inefficiency of traditional autoregressive visual generation by proposing RadAR, a radial parallel autoregressive framework that decodes tokens ring-by-ring from a center token. It preserves spatial locality through concentric rings and introduces a nested attention mechanism to mitigate inconsistent outputs, along with a tokenizer post-training step to align latent spaces. Empirically, RadAR achieves up to 5.6× speedups on ImageNet while delivering competitive image quality and enabling zero-shot generalization tasks such as out-painting and class-conditional editing, demonstrating strong practical impact for high-throughput visual synthesis and multimodal integration.

Abstract

Inspired by the remarkable success of autoregressive models in language modeling, this paradigm has been widely adopted in visual generation. However, the sequential token-by-token decoding mechanism inherent in traditional autoregressive models leads to low inference efficiency.In this paper, we propose RadAR, an efficient and parallelizable framework designed to accelerate autoregressive visual generation while preserving its representational capacity. Our approach is motivated by the observation that visual tokens exhibit strong local dependencies and spatial correlations with their neighbors--a property not fully exploited in standard raster-scan decoding orders. Specifically, we organize the generation process around a radial topology: an initial token is selected as the starting point, and all other tokens are systematically grouped into multiple concentric rings according to their spatial distances from this center. Generation then proceeds in a ring-wise manner, from inner to outer regions, enabling the parallel prediction of all tokens within the same ring. This design not only preserves the structural locality and spatial coherence of visual scenes but also substantially increases parallelization. Furthermore, to address the risk of inconsistent predictions arising from simultaneous token generation with limited context, we introduce a nested attention mechanism. This mechanism dynamically refines implausible outputs during the forward pass, thereby mitigating error accumulation and preventing model collapse. By integrating radial parallel prediction with dynamic output correction, RadAR significantly improves generation efficiency.

Figures (6)

• Figure 1: Generated samples from RadAR-XL. We show samples at different aspect ratios (top) and zero-shot image editing results (bottom) including class-conditional editing and out-painting . Please zoom in for details.
• Figure 2: The distribution of attention scores across visual tokens in different layers of the LlamaGen-XL sun2024autoregressive . Slash lines indicate that significant attention scores are allocated to tokens at fixed intervals, corresponding to tokens in the same column of previous rows.
• Figure 3: Illustration of our radial-parallel autoregressive (RadAR) generation framework. (a) Model implementation. RadAR takes the output $O_{k-1}$ from step $k-1$ and then uses learnable parameters to fill the range to be expanded outside $O_{k-1}$, forming the input $I_k$ for step $k$. For specific details on the extrapolation range, please refer to Algorithm 1. (b) Nested attention mask. For step $k$, green corresponds to the range where content has already been generated, and blue corresponds to the new unknown area expanded. The attention range is restricted to distinguish between the two purposes of error correction and content generation. Specifically, the newly expanded area in step $k$ accesses the context information of all tokens from previous $k-1$ steps and step $k$ to complete the generation purpose, while the generated area is restrict from accessing the expanded area.
• Figure 4: Visualization of error correction. Earlier generation errors are gradually corrected by the subsequent generation process as the number of inference steps increases. Please zoom in for details.
• Figure 5: Zero-shot evaluation in downstream tasks containing out-painting and class-conditional editing. The results show that RadAR can generalize to novel downstream tasks without special design and finetuning. Please zoom in for details.