ControlAR: Controllable Image Generation with Autoregressive Models

TL;DR

<3-5 sentence high-level summary> ControlAR tackles the challenge of adding fine-grained spatial control to autoregressive image generation, addressing encoding of 2D controls and per-token conditioning. It introduces a lightweight Vision Transformer-based control encoder to produce a sequence of control tokens and a unified conditional decoding scheme that fuses control tokens with image tokens during next-token prediction. The framework supports arbitrary-resolution generation via multi-resolution training (MR-ControlAR), maintaining efficiency while enabling outputs at varying sizes. Extensive experiments on class-to-image and text-to-image tasks with edges, depth, and segmentation masks show ControlAR achieving controllable, high-quality images and outperforming state-of-the-art controllable diffusion models like ControlNet++. Ablation studies validate design choices around the control encoder, fusion strategy, and training regimen, illustrating practical benefits for autoregressive models.

Abstract

Autoregressive (AR) models have reformulated image generation as next-token prediction, demonstrating remarkable potential and emerging as strong competitors to diffusion models. However, control-to-image generation, akin to ControlNet, remains largely unexplored within AR models. Although a natural approach, inspired by advancements in Large Language Models, is to tokenize control images into tokens and prefill them into the autoregressive model before decoding image tokens, it still falls short in generation quality compared to ControlNet and suffers from inefficiency. To this end, we introduce ControlAR, an efficient and effective framework for integrating spatial controls into autoregressive image generation models. Firstly, we explore control encoding for AR models and propose a lightweight control encoder to transform spatial inputs (e.g., canny edges or depth maps) into control tokens. Then ControlAR exploits the conditional decoding method to generate the next image token conditioned on the per-token fusion between control and image tokens, similar to positional encodings. Compared to prefilling tokens, using conditional decoding significantly strengthens the control capability of AR models but also maintains the model's efficiency. Furthermore, the proposed ControlAR surprisingly empowers AR models with arbitrary-resolution image generation via conditional decoding and specific controls. Extensive experiments can demonstrate the controllability of the proposed ControlAR for the autoregressive control-to-image generation across diverse inputs, including edges, depths, and segmentation masks. Furthermore, both quantitative and qualitative results indicate that ControlAR surpasses previous state-of-the-art controllable diffusion models, e.g., ControlNet++. Code, models, and demo will soon be available at https://github.com/hustvl/ControlAR.

Figures (16)

• Figure 1: Arbitrary-resolution images generated by ControlAR. Our ControlAR extends autoregressive models, e.g., LlamaGen llamagen, to generate high-quality images using spatial controls and expands the capability of autoregressive models to any-resolution image generation.
• Figure 2: Comparison between Conditional Prefilling v.s. Conditional Decoding. We encode the spatial control images into a sequence of control tokens for autoregressive models. (a) Conditional Prefilling: control condition tokens are prefilled into the autoregressive model before the first image token is generated. (b) Conditional Decoding: each image token is fused with the control condition token to predict the next image token. (c) Image Quality: we compare the performance (i.e., F1-Score and FID) across training epochs between conditional decoding and prefilling. It's remarkable that conditional decoding outperforms conditional prefilling in terms of performance and training convergence speed. (d) Training cost: conditional prefilling significantly increases the training memory (+59.1%) and training latency (+96.3%) compared to conditional decoding.
• Figure 3: The overall architecture of ControlAR. The control image will be flattened into patches and encoded as a sequence of control tokens via the proposed control encoder. For controllable image generation, we extend several sequential layers (i.e., causal Transformer layer or Mamba layer) of the autoregressive model into conditional sequential layers by incorporating the fusion of control tokens and image tokens to predict the next image token. Finally, the image tokens are decoded into a generated image through the VQGAN decoder.
• Figure 4: Visualization of C2I controllable generation. Our ControlAR generates images with high conditional consistency and quality on both LlamaGen and AiM.
• Figure 5: Visualization of text-to-image controllable generation. We use red boxes to mark areas where the generated results of other methods differ from the input control image.