CAR: Controllable Autoregressive Modeling for Visual Generation

TL;DR

<3-5 sentence high-level summary> CAR introduces Controllable AutoRegressive Modeling (CAR), a plug-and-play framework that injects multi-scale control representations into a frozen pre-trained autoregressive visual model to achieve fine-grained controllable image generation with significantly reduced training resources. By leveraging a next-scale VAR foundation and a parallel control branch, CAR models p(I|C) through hierarchical token maps and Bayesian-inspired posterior optimization, enabling robust generalization to unseen categories. Quantitative and qualitative results on ImageNet show CAR surpasses diffusion-based baselines in image quality and runs more efficiently, with user studies confirming higher perceived controllability. The work demonstrates a scalable pathway to end-to-end controllable generation for autoregressive vision models, highlighting strong cross-domain generalization and practicality for large-scale applications.

Abstract

Controllable generation, which enables fine-grained control over generated outputs, has emerged as a critical focus in visual generative models. Currently, there are two primary technical approaches in visual generation: diffusion models and autoregressive models. Diffusion models, as exemplified by ControlNet and T2I-Adapter, offer advanced control mechanisms, whereas autoregressive models, despite showcasing impressive generative quality and scalability, remain underexplored in terms of controllability and flexibility. In this study, we introduce Controllable AutoRegressive Modeling (CAR), a novel, plug-and-play framework that integrates conditional control into multi-scale latent variable modeling, enabling efficient control generation within a pre-trained visual autoregressive model. CAR progressively refines and captures control representations, which are injected into each autoregressive step of the pre-trained model to guide the generation process. Our approach demonstrates excellent controllability across various types of conditions and delivers higher image quality compared to previous methods. Additionally, CAR achieves robust generalization with significantly fewer training resources compared to those required for pre-training the model. To the best of our knowledge, we are the first to propose a control framework for pre-trained autoregressive visual generation models.

Figures (6)

• Figure 1: Controllable generation using CAR under various conditions. Results are $512\times 512$.
• Figure 2: Overview of the proposed Controllable AutoRegressive Modeling (CAR) framework. CAR integrates multi-scale latent variable modeling, where control representation is progressively refined and injected into the generation process of a pre-trained autoregressive model. Previous image tokens are accumulated and upsampled to form $b_k$, which serves as the input token for scale $k$. Each scale's token map $r_k$ is predicted based on previous tokens and the corresponding control input $c_k$, ensuring that the generated image $\mathcal{I}$ adheres to the specified visual conditions $\mathcal{C}$.
• Figure 3: Scaling laws of our CAR model. It can be observed that as the model depth increases, the four image quality metrics improve across the five conditions.
• Figure 4: Results are presented for five distinct conditions, where the top row shows the input conditions, and the following rows display the generated images. These categories are excluded from the training set, demonstrating that the CAR learns the general semantics from the input conditions.
• Figure 5: T-SNE visualization of the distribution of generation results from our CAR model and the vanilla model.