Customize Your Visual Autoregressive Recipe with Set Autoregressive Modeling

TL;DR

This paper introduces Set AutoRegressive Modeling (SAR), a generalization of autoregressive image generation that permits arbitrary sequences of token-sets and employs a Fully Masked Transformer to handle generalized causal masks. By unifying AR, MAR, and VAR as special cases, SAR enables smooth transitions between inference regimes and supports both few-step generation and KV-cache acceleration, demonstrated on ImageNet and a high-capacity 900M text-to-image model (Lumina-SAR) capable of high-resolution outputs. The authors provide extensive experiments on sequence-order and interval configurations, ablations on training strategies, and an impactful application to zero-shot text-to-image generation and editing. Overall, SAR offers a flexible, causality-preserving framework that can adapt AR-based modeling to diverse modalities and efficiency requirements.

Abstract

We introduce a new paradigm for AutoRegressive (AR) image generation, termed Set AutoRegressive Modeling (SAR). SAR generalizes the conventional AR to the next-set setting, i.e., splitting the sequence into arbitrary sets containing multiple tokens, rather than outputting each token in a fixed raster order. To accommodate SAR, we develop a straightforward architecture termed Fully Masked Transformer. We reveal that existing AR variants correspond to specific design choices of sequence order and output intervals within the SAR framework, with AR and Masked AR (MAR) as two extreme instances. Notably, SAR facilitates a seamless transition from AR to MAR, where intermediate states allow for training a causal model that benefits from both few-step inference and KV cache acceleration, thus leveraging the advantages of both AR and MAR. On the ImageNet benchmark, we carefully explore the properties of SAR by analyzing the impact of sequence order and output intervals on performance, as well as the generalization ability regarding inference order and steps. We further validate the potential of SAR by training a 900M text-to-image model capable of synthesizing photo-realistic images with any resolution. We hope our work may inspire more exploration and application of AR-based modeling across diverse modalities.

Figures (16)

• Figure 1: Text-conditioned and class-conditioned samples generated by SAR models. Our T2I model generates $1024\times1024$ images at a speed $60$ times faster than AR models.
• Figure 2: Conceptual illustration. SAR integrates existing AR variants by manipulating the sequence order and output intervals, creating a smooth transition path from classic AR to MAR.
• Figure 3: The model architecture of Fully Masked Transformer. Conceptually, it is the transformer in vaswani2017attention plus generalized causal masks.
• Figure 4: Some sequence order settings in the experiment. Taking the $8\times8$ case as illustration.
• Figure 5: Trade-off between performance and time, using LlamaGen-L as a reference.