D-AR: Diffusion via Autoregressive Models

TL;DR

D-AR introduces a diffusion-to-autoregressive bridge for visual generation by a sequential diffusion tokenizer that converts images into a 1D token sequence of length $N$ (e.g., $N=256$) representing progressive diffusion steps. An off-the-shelf decoder-only Transformer then performs vanilla next-token prediction on these tokens, while diffusion steps are realized through token-conditioned sampling and flow matching, preserving the standard AR objective. The method yields competitive ImageNet 256x256 results, with D-AR-XL (775M) achieving around $2.09$ FID and token reconstructions attaining $rFID\approx1.52$, while offering practical benefits such as KV-cache-friendly inference, streaming previews, and zero-shot layout-controlled synthesis via prefix tokens. By keeping the AR framework intact and delegating diffusion dynamics to a diffusion decoder within the tokenizer, D-AR showcases a unified approach that leverages large language model-style architectures for native visual generation with diffusion-informed conditioning.

Abstract

This paper presents Diffusion via Autoregressive models (D-AR), a new paradigm recasting the image diffusion process as a vanilla autoregressive procedure in the standard next-token-prediction fashion. We start by designing the tokenizer that converts images into sequences of discrete tokens, where tokens in different positions can be decoded into different diffusion denoising steps in the pixel space. Thanks to the diffusion properties, these tokens naturally follow a coarse-to-fine order, which directly lends itself to autoregressive modeling. Therefore, we apply standard next-token prediction on these tokens, without modifying any underlying designs (either causal masks or training/inference strategies), and such sequential autoregressive token generation directly mirrors the diffusion procedure in image space. That is, once the autoregressive model generates an increment of tokens, we can directly decode these tokens into the corresponding diffusion denoising step in the streaming manner. Our pipeline naturally reveals several intriguing properties, for example, it supports consistent previews when generating only a subset of tokens and enables zero-shot layout-controlled synthesis. On the standard ImageNet benchmark, our method achieves 2.09 FID using a 775M Llama backbone with 256 discrete tokens. We hope our work can inspire future research on unified autoregressive architectures of visual synthesis, especially with large language models. Code and models will be available at https://github.com/showlab/D-AR

Figures (13)

• Figure 1: Uncurated generated samples from D-AR-XL with $256\times 256$ resolutions (CFG=4.0).
• Figure 2: Different paradigms incorporating diffusion and autoregressive models for vision generation. (a) uses patch-level diffusion during every single autoregressive step to tackle continuous outputs marcausalfusiondiffusionforcing; (b) DART dart denoises a full image per every autoregressive step by AR transformer, together inputted with history denoised images; (c) use a single set of continuous outputs to different diffusion steps and require diffusion gradients to train AR models dreamllmseedxnextgptmetaquery; (d) ours uses vanilla AR models, which can train with discrete inputs/outputs by simply cross entropy, and sequentially decode output tokens with our diffusion tokenizer.
• Figure 3: Sequential diffusion tokenizer structure. When training the tokenizer, the pixel diffusion transformer in the tokenizer decoder calculates the velocity loss with the selected group of tokens, $\mathbf{c}(t)$, as conditioning tokens.
• Figure 4: Reconstruction results on ImageNet validation 50K samples with 256 discrete tokens. We also finetune our sequential diffusion tokenizer with smaller codebook size, 4096, and compare with LlamaGen tokenizer counterpart.
• Figure 5: Different sampling configurations on our sequential diffusion tokenizer. Adams 2nd refers to the two step Adams–Bashforth solver adams2nd, while others use Euler.