Context-Aware Autoregressive Models for Multi-Conditional Image Generation
Yixiao Chen, Zhiyuan Ma, Guoli Jia, Che Jiang, Jianjun Li, Bowen Zhou
TL;DR
ContextAR introduces a unified, context-aware autoregressive framework for multi-conditional image generation, addressing the rigidity of diffusion-based methods in handling heterogeneous conditions. By embedding diverse conditions as tokens in a single sequence and employing hybrid positional embeddings (RoPE+LPE) along with Cross-Condition Perception Restriction and Intra-Condition Bidirectional Perception, ContextAR enables flexible, zero-shot combination of conditions during inference. The approach achieves competitive image quality and superior controllability across datasets (e.g., MultiGen-20M, SubjectSpatial200K) compared with diffusion-based and autoregressive baselines, while reducing computational complexity and enabling scalable multi-condition control. These findings demonstrate the practical impact of unified multimodal autoregressive modeling for fine-grained, multi-condition image synthesis in real-world applications, with potential for broader multimodal generation tasks. ${p}({\mathbf{q}} \mid c) = \prod_{t=1}^{N} p(q_t \mid q_{<t}, c)$ and related training/inference strategies underpin the learning dynamics that tie together diverse visual and textual conditions.$
Abstract
Autoregressive transformers have recently shown impressive image generation quality and efficiency on par with state-of-the-art diffusion models. Unlike diffusion architectures, autoregressive models can naturally incorporate arbitrary modalities into a single, unified token sequence--offering a concise solution for multi-conditional image generation tasks. In this work, we propose $\textbf{ContextAR}$, a flexible and effective framework for multi-conditional image generation. ContextAR embeds diverse conditions (e.g., canny edges, depth maps, poses) directly into the token sequence, preserving modality-specific semantics. To maintain spatial alignment while enhancing discrimination among different condition types, we introduce hybrid positional encodings that fuse Rotary Position Embedding with Learnable Positional Embedding. We design Conditional Context-aware Attention to reduces computational complexity while preserving effective intra-condition perception. Without any fine-tuning, ContextAR supports arbitrary combinations of conditions during inference time. Experimental results demonstrate the powerful controllability and versatility of our approach, and show that the competitive perpormance than diffusion-based multi-conditional control approaches the existing autoregressive baseline across diverse multi-condition driven scenarios. Project page: $\href{https://context-ar.github.io/}{https://context-ar.github.io/.}$