Visual Autoregressive Modeling for Instruction-Guided Image Editing

TL;DR

This paper addresses the limitations of diffusion-based instruction-guided image editing, notably unintended edits and slow inference, by introducing VAREdit, a visual autoregressive framework that casts editing as next-scale prediction. Built on a pre-trained VAR backbone, VAREdit generates multi-scale target residuals conditioned on a source image and textual instruction, using a Scale-Aligned Reference (SAR) to inject scale-matched conditioning in the first self-attention layer. The approach achieves state-of-the-art GPT-Balance and CLIP-based metrics on EMU-Edit and PIE-Bench, with substantial speedups (e.g., 512×512 edits in 1.2 seconds, 2.2× faster than comparable methods). Empirical results from ablations validate the SAR design and model scaling (2B vs 8B), demonstrating strong editing fidelity, robustness across editing types, and superior efficiency, signaling a promising direction for efficient, high-fidelity AR-based image editing.

Abstract

Recent advances in diffusion models have brought remarkable visual fidelity to instruction-guided image editing. However, their global denoising process inherently entangles the edited region with the entire image context, leading to unintended spurious modifications and compromised adherence to editing instructions. In contrast, autoregressive models offer a distinct paradigm by formulating image synthesis as a sequential process over discrete visual tokens. Their causal and compositional mechanism naturally circumvents the adherence challenges of diffusion-based methods. In this paper, we present VAREdit, a visual autoregressive (VAR) framework that reframes image editing as a next-scale prediction problem. Conditioned on source image features and text instructions, VAREdit generates multi-scale target features to achieve precise edits. A core challenge in this paradigm is how to effectively condition the source image tokens. We observe that finest-scale source features cannot effectively guide the prediction of coarser target features. To bridge this gap, we introduce a Scale-Aligned Reference (SAR) module, which injects scale-matched conditioning information into the first self-attention layer. VAREdit demonstrates significant advancements in both editing adherence and efficiency. On EMU-Edit and PIE-Bench benchmarks, VAREdit outperforms leading diffusion-based methods by a substantial margin in terms of both CLIP and GPT scores. Moreover, VAREdit completes a 512$\times$512 editing in 1.2 seconds, making it 2.2$\times$ faster than the similarly sized UltraEdit. Code is available at: https://github.com/HiDream-ai/VAREdit.

Figures (17)

• Figure 1: VAREdit achieves high-precision performance in instruction-guided image editing. It excels across diverse editing scenarios, including object-level modifications (addition, replacement, removal), attribute changes (material, text, posture, style, color) and complex compositional edits.
• Figure 2: The overall architecture of VAREdit for instruction-guided image editing. VAREdit first encodes the images into multi-scale residuals by a shared vector quantizer and maps the instructions into textual token embeddings. These features are organized as the finest-scale source feature $\mathbf{F}_{K}^{(src)}$, the pooled textual representation $\widetilde{\mathbf{F}}_0^{(tgt)}$, the coarse-to-fine target features $\widetilde{\mathbf{F}}_{1:K-1}^{(tgt)}$, and then sent to the VAR Transformer. The source feature $\mathbf{F}_{K}^{(src)}$ is further sent to the Scale-Aligned Reference (SAR) module in the first self-attention layer, while the textual token embeddings are also used for cross-attention calculations of key and value matrices. The ground truth residuals $\mathbf{R}_{1:K}^{(tgt)}$ guide the training of the last $K$ output residuals $\hat{\mathbf{R}}_{1:K}^{(tgt)}$. During inference, the residuals $\hat{\mathbf{R}}_{1:K}^{(tgt)}$ are predicted autoregressively, which are then aggregated to $\hat{\mathbf{F}}_K^{(tgt)}$ and then decoded to the edited image.
• Figure 3: Self-attention heatmaps of PIE-Bench samples based on the full-scale transformer across different layers. The dependency patterns differ in the first self-attention layer versus others, inspiring a scale-aligned reference module.
• Figure 4: Scatter plots of successful editing samples capacity versus the GPT-Over. scores over the successful editing subsets of basic approaches on EMU-Edit and PIE-Bench benchmarks.
• Figure 5: Fine-grained categorical GPT-Suc. and GPT-Bal. scores of VAREdit and basic approaches on EMU-Edit and PIE-Bench benchmarks.