EditAR: Unified Conditional Generation with Autoregressive Models

TL;DR

This work proposes EditAR, a single unified autoregressive framework for a variety of conditional image generation tasks, e.g., image editing, depth-to-image, edge-to-image, segmentation-to-image, and evaluates its effectiveness across diverse tasks on established benchmarks.

Abstract

Recent progress in controllable image generation and editing is largely driven by diffusion-based methods. Although diffusion models perform exceptionally well in specific tasks with tailored designs, establishing a unified model is still challenging. In contrast, autoregressive models inherently feature a unified tokenized representation, which simplifies the creation of a single foundational model for various tasks. In this work, we propose EditAR, a single unified autoregressive framework for a variety of conditional image generation tasks, e.g., image editing, depth-to-image, edge-to-image, segmentation-to-image. The model takes both images and instructions as inputs, and predicts the edited images tokens in a vanilla next-token paradigm. To enhance the text-to-image alignment, we further propose to distill the knowledge from foundation models into the autoregressive modeling process. We evaluate its effectiveness across diverse tasks on established benchmarks, showing competitive performance to various state-of-the-art task-specific methods. Project page: https://jitengmu.github.io/EditAR/

Figures (14)

• Figure 1: We propose EditAR, a unified conditional autoregressive model for diverse conditional generation tasks. We demonstrate that without task-specific designs, a single autoregressive model achieves strong performance across diverse tasks, including texture manipulation, object replacement, object removal, local editing, canny-to-image, depth-to-image, and segmentation-to-image.
• Figure 2: Overview of EditAR, which can take various types of image conditions to perform image editing or translation. An image $\mathcal{I}_c$ is mapped through a VQ-Encoder $\mathcal{E}_{\mathcal{I}}$ to obtain corresponding token indices. Corresponding text instructions are mapped to latent embeddings $c_{\mathcal{T}}$ via a text encoder $\mathcal{E}_{\mathcal{T}}$. Both image token indices and text embeddings are input to the autoregressive transformer $\mathcal{F}$ to predict the target token indices $s$. To enhance the text-to-image alignment, a distillation loss is introduced during training to minimize the differences between the latent features of the autoregressive model, $\mathcal{F}$ and that of a feature encoder $\mathcal{E}_{distill}$. The output sequence $s$ is lastly decoded into a realistic image via a VQ-Decoder $\mathcal{D}_{\mathcal{I}}$ during inference.
• Figure 3: Comparison of EditAR (Ours) to feed-forward methods (InstructPix2Pix brooks2022instructpix2pix, MGIE fu2024mgie) and inversion-based approaches (Prompt-to-Prompt hertz2022prompt, PnPInversion ju2023pnpinversion) on various edits. Our method preserves input details well and has strong text-to-image alignment. In contrast, baseline results exhibit unrealistic visuals, including exaggerated edits or not following instructions, unrealistic modifications, or are unable to localize objects accurately.
• Figure 4: Visual comparisons to baseline methods on various image translation tasks. Our method, EditAR, produces photo-realistic results, preserves input details, and offers substantial sample diversity.
• Figure 5: Studies on distillation loss. From left to right, we show the input image, results w/out distillation, and distillation results with DINOv2 and CLIP. The top example shows improved object localization. The bottom shows better text-to-image alignment.