Continuous Visual Autoregressive Generation via Score Maximization

TL;DR

The paper tackles the challenge of autoregressive generation for continuous visual data by eliminating vector quantization and framing training as maximizing strictly proper scoring rules. It introduces Continuous VAR and, in particular, the energy-based Energy Transformer (EAR), which uses an energy score as a likelihood-free objective and a sampling-based MLP generator to produce continuous tokens. The approach unifies prior continuous-generation methods under score-maximization and demonstrates competitive ImageNet 256×256 results with superior inference efficiency compared to diffusion and discrete-token baselines. By illustrating the importance of strict propriety and expressiveness, the work highlights practical gains in reconstruction quality and speed, and lays out clear directions for extending continuous autoregression to broader modalities and scores.

Abstract

Conventional wisdom suggests that autoregressive models are used to process discrete data. When applied to continuous modalities such as visual data, Visual AutoRegressive modeling (VAR) typically resorts to quantization-based approaches to cast the data into a discrete space, which can introduce significant information loss. To tackle this issue, we introduce a Continuous VAR framework that enables direct visual autoregressive generation without vector quantization. The underlying theoretical foundation is strictly proper scoring rules, which provide powerful statistical tools capable of evaluating how well a generative model approximates the true distribution. Within this framework, all we need is to select a strictly proper score and set it as the training objective to optimize. We primarily explore a class of training objectives based on the energy score, which is likelihood-free and thus overcomes the difficulty of making probabilistic predictions in the continuous space. Previous efforts on continuous autoregressive generation, such as GIVT and diffusion loss, can also be derived from our framework using other strictly proper scores. Source code: https://github.com/shaochenze/EAR.

Figures (8)

• Figure 1: Comparison between the discrete-token standard Transformer and our continuous-token energy Transformer. At the input side, the embedding lookup table is replaced with a linear projection. At the output side, the softmax classification layer is replaced with a small MLP generator, which takes random noise $\epsilon$ as input to perturb the hidden state.
• Figure 2: The speed/quality trade-off for EAR and MAR. The number of autoregressive steps is fixed at 64. For MAR, we vary the number of diffusion steps (10, 20, 25, 30, 40, 50) to generate outputs under different inference latencies. For EAR, the curve is obtained by using different model sizes (EAR-B, EAR-L, EAR-H). The inference time is measured on a single A100 GPU.
• Figure 3: FID curves of the continuous-valued energy Transformer (205M) and the discrete-valued standard Transformer (196M). The guidance scale is 3.0.
• Figure 4: Generation quality of the Gaussian Transformer under different standard deviations during inference. The model size is 184M. cfg is disabled since it does not work well here.
• Figure 5: The results of varying learning rates for EAR-B.