Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nested AutoRegressive Models

Hongyu Wu, Xuhui Fan, Zhangkai Wu, Longbing Cao

TL;DR

NestAR tackles inefficiency and limited diversity in autoregressive image generation by introducing a nested, two-level AR architecture with hierarchical multi-scale modules that generate image patches. Each scale uses an AR process to produce patches conditioned on outputs from earlier scales, and an internal AR generates within-patch tokens, reducing complexity from $O(n)$ to $O(\log n)$. Training combines flow matching with a velocity-coordination objective to align module behaviors and employs a continuous-token framework. Empirical results on ImageNet-256 show NestAR achieves a top IS score and competitive FID while offering substantial inference speedups over diffusion and other AR-based methods, and improved sample diversity. This approach provides a practical path toward fast, diverse, high-fidelity AR-based image synthesis.

Abstract

AutoRegressive (AR) models have demonstrated competitive performance in image generation, achieving results comparable to those of diffusion models. However, their token-by-token image generation mechanism remains computationally intensive and existing solutions such as VAR often lead to limited sample diversity. In this work, we propose a Nested AutoRegressive~(NestAR) model, which proposes nested AutoRegressive architectures in generating images. NestAR designs multi-scale modules in a hierarchical order. These different scaled modules are constructed in an AR architecture, where one larger-scale module is conditioned on outputs from its previous smaller-scale module. Within each module, NestAR uses another AR structure to generate ``patches'' of tokens. The proposed nested AR architecture reduces the overall complexity from $\mathcal{O}(n)$ to $\mathcal{O}(\log n)$ in generating $n$ image tokens, as well as increases image diversities. NestAR further incorporates flow matching loss to use continuous tokens, and develops objectives to coordinate these multi-scale modules in model training. NestAR achieves competitive image generation performance while significantly lowering computational cost.

Nested AutoRegressive Models

TL;DR

NestAR tackles inefficiency and limited diversity in autoregressive image generation by introducing a nested, two-level AR architecture with hierarchical multi-scale modules that generate image patches. Each scale uses an AR process to produce patches conditioned on outputs from earlier scales, and an internal AR generates within-patch tokens, reducing complexity from to . Training combines flow matching with a velocity-coordination objective to align module behaviors and employs a continuous-token framework. Empirical results on ImageNet-256 show NestAR achieves a top IS score and competitive FID while offering substantial inference speedups over diffusion and other AR-based methods, and improved sample diversity. This approach provides a practical path toward fast, diverse, high-fidelity AR-based image synthesis.

Abstract

AutoRegressive (AR) models have demonstrated competitive performance in image generation, achieving results comparable to those of diffusion models. However, their token-by-token image generation mechanism remains computationally intensive and existing solutions such as VAR often lead to limited sample diversity. In this work, we propose a Nested AutoRegressive~(NestAR) model, which proposes nested AutoRegressive architectures in generating images. NestAR designs multi-scale modules in a hierarchical order. These different scaled modules are constructed in an AR architecture, where one larger-scale module is conditioned on outputs from its previous smaller-scale module. Within each module, NestAR uses another AR structure to generate ``patches'' of tokens. The proposed nested AR architecture reduces the overall complexity from to in generating image tokens, as well as increases image diversities. NestAR further incorporates flow matching loss to use continuous tokens, and develops objectives to coordinate these multi-scale modules in model training. NestAR achieves competitive image generation performance while significantly lowering computational cost.
Paper Structure (21 sections, 7 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 21 sections, 7 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. AutoRegressive Model
  4. AutoRegressive Model for Images
  5. Diffusion and Flow Matching Models
  6. Methodology
  7. Nested AutoRegressive Model
  8. Two Levels of AR Structures in NestAR.
  9. Calculating the Conditional Distribution
  10. Coordinating Scaled Modules
  11. Image Generation in NestAR
  12. Connections to Existing AR Models
  13. Experiments
  14. Experimental Settings
  15. Encoder and Decoder.
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Visual comparison between VAR (left panel) and the proposed NestAR (right panel). VAR: next resolution prediction from coarse to fine resolutions of the entire image. A single AR model all $K$ resolutions. (b) NestAR with $3$ scale modules: different scaled modules generating progressive larger area of the image. These modules are bounded by red, black, and purple boxes correspondingly.
  • Figure 2: Visualization of the mechanisms for different AR models. (a), NestAR model; (b), vanilla AR model; (c) Visual AutoRegressive Model (VAR). NestAR expands the sizes of the patches along with the module orders. The $1$-st scaled module, which is also the smallest, captures the distribution of the smallest patch of tokens. Its output then becomes the first patch of tokens for the $2$-nd scaled module which models a larger-sized patch of tokens. This process continues to the highest scaled-module, in which the generated patches can form the entire image. Vanilla AR generates tokens one at a time based on previous tokens. VAR generates different resolutions of the entire image in a hierarchical manner.
  • Figure 3: Qualitative Results: Generated $256\times 256$ image samples from our NestAR-H model.
  • Figure 4: Qualitative Results: $256\times 256$ image samples of the same classes to demonstrate diversity of images. The classes from left to right are: Daisy, Volcano, Alps, and Coral.