M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation

TL;DR

M-VAR introduces a decoupled approach to scale-wise autoregressive image generation by separating intra-scale (bidirectional) attention from inter-scale (coarse-to-fine) dependencies modeled with Mamba, a linear-complexity state-space method. This design preserves 2D spatial coherence while dramatically reducing computation, yielding higher image fidelity with fewer parameters and faster inference than prior VAR models. Empirical results on ImageNet demonstrate state-of-the-art FID/IS trade-offs at 256×256 and 512×512 resolutions, with the largest M-VAR-d32 achieving 1.78 FID on 256×256 and outperforming diffusion and autoregressive baselines. The work also provides thorough ablations validating the decoupled architecture and highlights practical gains such as faster inference and parameter efficiency, making high-quality autoregressive image generation more scalable.

Abstract

There exists recent work in computer vision, named VAR, that proposes a new autoregressive paradigm for image generation. Diverging from the vanilla next-token prediction, VAR structurally reformulates the image generation into a coarse to fine next-scale prediction. In this paper, we show that this scale-wise autoregressive framework can be effectively decoupled into \textit{intra-scale modeling}, which captures local spatial dependencies within each scale, and \textit{inter-scale modeling}, which models cross-scale relationships progressively from coarse-to-fine scales. This decoupling structure allows to rebuild VAR in a more computationally efficient manner. Specifically, for intra-scale modeling -- crucial for generating high-fidelity images -- we retain the original bidirectional self-attention design to ensure comprehensive modeling; for inter-scale modeling, which semantically connects different scales but is computationally intensive, we apply linear-complexity mechanisms like Mamba to substantially reduce computational overhead. We term this new framework M-VAR. Extensive experiments demonstrate that our method outperforms existing models in both image quality and generation speed. For example, our 1.5B model, with fewer parameters and faster inference speed, outperforms the largest VAR-d30-2B. Moreover, our largest model M-VAR-d32 impressively registers 1.78 FID on ImageNet 256$\times$256 and outperforms the prior-art autoregressive models LlamaGen/VAR by 0.4/0.19 and popular diffusion models LDM/DiT by 1.82/0.49, respectively. Code is avaiable at \url{https://github.com/OliverRensu/MVAR}.

Figures (5)

• Figure 1: Generated 512$\times$512 and 256$\times$256 samples from our M-VAR trained on ImageNet.
• Figure 2: Fréchet inception distance (FID) on 256$\times$256 image generation. Our M-VAR-1.5B model outperforms the largest 2B VAR-d30 with fewer parameters and faster inference speed. Our largest M-VAR-3B achieves 1.78 FID.
• Figure 3: An overview of M-VAR. M-VAR takes the input sequence of $\{[C], s_1, ..., s_{n-1}\}$ to predict $\{s_1, ..., s_{n}\}$ where $[C]$ is the condition token. The model first divides the input into different scales and applies the standard attention mechanism to capture intra-scale spatial correlations. It then utilizes Mamba to autoregressively model inter-scale dependencies, enabling coherent and efficient multi-scale image generation.
• Figure 4: Qualitative Results. We show the images generated by our M-VAR.
• Figure 5: The effectiveness of our decouple design. We gradually replace the global attention with our intra-scale attention and inter-scale mamba.