Hyperspherical Latents Improve Continuous-Token Autoregressive Generation

TL;DR

The theoretical analysis shows that hyperspherical constraint removes the scale component (the primary cause of variance collapse), thereby stabilizing AR decoding, and this is the first time a pure next-token AR image generator with raster order surpasses diffusion and masked-generation models at comparable parameter scales.

Abstract

Autoregressive (AR) models are promising for image generation, yet continuous-token AR variants often trail latent diffusion and masked-generation models. The core issue is heterogeneous variance in VAE latents, which is amplified during AR decoding, especially under classifier-free guidance (CFG), and can cause variance collapse. We propose SphereAR to address this issue. Its core design is to constrain all AR inputs and outputs -- including after CFG -- to lie on a fixed-radius hypersphere (constant $\ell_2$ norm), leveraging hyperspherical VAEs. Our theoretical analysis shows that hyperspherical constraint removes the scale component (the primary cause of variance collapse), thereby stabilizing AR decoding. Empirically, on ImageNet generation, SphereAR-H (943M) sets a new state of the art for AR models, achieving FID 1.34. Even at smaller scales, SphereAR-L (479M) reaches FID 1.54 and SphereAR-B (208M) reaches 1.92, matching or surpassing much larger baselines such as MAR-H (943M, 1.55) and VAR-d30 (2B, 1.92). To our knowledge, this is the first time a pure next-token AR image generator with raster order surpasses diffusion and masked-generation models at comparable parameter scales.

Figures (21)

• Figure 1: Left: FID vs. parameters on ImageNet 256$\times$256 class-conditional generation, SphereAR attains lower FID with fewer parameters. Right: 256$\times$256 samples generated by SphereAR-L (479M).
• Figure 2: Overview of SphereAR. Left: A hyperspherical VAE (S-VAE) encodes raw data into a sequence of latent tokens constrained to a fixed-radius hypersphere $\mathbb{S}^{d-1}$. The encoder outputs a unit mean direction $\boldsymbol{\mu}$ and a concentration $\kappa$ that parameterize a von Mises--Fisher (vMF) or Power Spherical posterior. Right: A causal Transformer with a token-level diffusion head models the next-token distribution over the hyperspherical token sequence. At inference, the AR model’s predictions, including CFG-rescaled ones, are projected back onto the fixed-radius hypersphere. The VAE decoder then reconstructs the image from the predicted hyperspherical latents.
• Figure 3: Visualization of token distributions. Each panel shows one token type, with three tokens in different colors. (a) Discrete tokens lie on the probability simplex and are intrinsically scale-invariant. (b) Diagonal-Gaussian latents are unconstrained in scale; despite a KL prior, per-dimension/token variances remain heterogeneous. (c) Hyperspherical latents constrain each token to a fixed norm (e.g., $\lVert\mathbf{z}\rVert_2=R$), yielding scale-invariant representations. In practice, (a) and (c) are robust under AR decoding, whereas (b) is prone to scale drift and occasional variance collapse (e.g., with CFG).
• Figure 4: Impact of VAE variants on generation performance (FID vs. CFG). All variants share the same backbone and training/evaluation setup; only the VAE objective/posterior differs. Left: diagonal-Gaussian with enlarged KL weight (G-01/04/08/16), $\sigma$-VAE with fixed scale (F-01/02/05), and S-VAE with a Power Spherical posterior (S-01/04/08). Right: additionally includes diagonal-Gaussian with post-hoc normalization (N-01/04/08/16).
• Figure 5: Directional distributions on the sphere. Left: Power Spherical respects purely directional geometry—the density depends only on $\boldsymbol{\mu}^{\top}\mathbf{u}$ and is axially symmetric, with a single concentration parameter $\kappa$. Right: Gaussian$+$norm induces a projected-normal (ACG) law whose level sets follow $\mathbf{u}^{\top}\boldsymbol{\Sigma}^{-1}\mathbf{u}$; symmetry axes are determined by $\boldsymbol{\Sigma}$ (and the Gaussian mean), so the density is typically elliptical rather than axially symmetric.