Resurrect Mask AutoRegressive Modeling for Efficient and Scalable Image Generation

TL;DR

This work addresses the efficiency gap in image generation by resurrecting Mask AutoRegressive (MAR) models through careful tokenizer selection and architectural redesign. It introduces MaskGIL, a Bidirectional LLaMA-based MAR model with 2D RoPE, stabilized by QK-Norm and Post-Norm, achieving competitive or state-of-the-art quality with as few as 8 inference steps. The authors scale MaskGIL from 111M to 1.4B and extend to text-driven generation (775M) and practical applications, including a unified AR-MAR inference framework and a real-time speech-to-image system. The study demonstrates that combining AR-driven prompts with MAR completion yields substantial speedups while preserving fidelity, expanding MAR's applicability beyond static image generation to real-time and multimodal scenarios.

Abstract

AutoRegressive (AR) models have made notable progress in image generation, with Masked AutoRegressive (MAR) models gaining attention for their efficient parallel decoding. However, MAR models have traditionally underperformed when compared to standard AR models. This study refines the MAR architecture to improve image generation quality. We begin by evaluating various image tokenizers to identify the most effective one. Subsequently, we introduce an improved Bidirectional LLaMA architecture by replacing causal attention with bidirectional attention and incorporating 2D RoPE, which together form our advanced model, MaskGIL. Scaled from 111M to 1.4B parameters, MaskGIL achieves a FID score of 3.71, matching state-of-the-art AR models in the ImageNet 256x256 benchmark, while requiring only 8 inference steps compared to the 256 steps of AR models. Furthermore, we develop a text-driven MaskGIL model with 775M parameters for generating images from text at various resolutions. Beyond image generation, MaskGIL extends to accelerate AR-based generation and enable real-time speech-to-image conversion. Our codes and models are available at https://github.com/synbol/MaskGIL.

Figures (10)

• Figure 1: Image Generation with our MaskGIL Model and Unified Inference Framework. We show samples from our class-driven generation (top) and text-driven generation (middle) in various resolutions. At the bottom, we show the generation results of MaskGIL fused with Lumina-mGPT.
• Figure 2: Illustration of (a) discrete-value image tokenizer (encoder and quantizer) and decoder via image reconstruction, (b) training the AR model through causal attention modeling and (c) training the MAR model through bidirectional attention modeling.
• Figure 3: Visual Generation Evalutation. We show FID of class-driven ImageNet 256$\times$256 benchmark over training epochs.
• Figure 4: The Effect of Model Size, CFG, QK-Norm, and Post-Norm. We show the FID scores on the ImageNet benchmark across different model sizes and CFG configurations. Scaling the model size consistently improves FID scores throughout the training process. The impact of CFG is also notable. To monitor training stability, we plot the model’s output norm.
• Figure 5: Performance Comparison of Different Decoding Steps and Different Schedulers. This experiment shows the checkpoint results of MaskGIL-B at 400 epochs, with CFG$=2.0$.