EditInfinity: Image Editing with Binary-Quantized Generative Models

TL;DR

EditInfinity tackles text-driven image editing by leveraging Infinity, a binary-quantized autoregressive model, to enable exact supervision during image inversion and robust holistic smoothing during editing. Inversion is formulated as optimizing a textual embedding aligned to the source image using exact quantized tokens $R_{1...K}^{sou}$, augmented with text-prompt rectification and LoRA-based style preservation. Editing employs a piecewise linear smoothing kernel $ ext{G}$ and multi-scale autoregressive token blending to achieve precise semantic edits with faithful background preservation. On PIE-Bench, EditInfinity achieves state-of-the-art trade-offs between background fidelity and text alignment, delivering strong quantitative and user-study results while enabling faster iterative editing after a single inversion, demonstrating the practicality and effectiveness of binary-quantized models for image editing.

Abstract

Adapting pretrained diffusion-based generative models for text-driven image editing with negligible tuning overhead has demonstrated remarkable potential. A classical adaptation paradigm, as followed by these methods, first infers the generative trajectory inversely for a given source image by image inversion, then performs image editing along the inferred trajectory guided by the target text prompts. However, the performance of image editing is heavily limited by the approximation errors introduced during image inversion by diffusion models, which arise from the absence of exact supervision in the intermediate generative steps. To circumvent this issue, we investigate the parameter-efficient adaptation of binary-quantized generative models for image editing, and leverage their inherent characteristic that the exact intermediate quantized representations of a source image are attainable, enabling more effective supervision for precise image inversion. Specifically, we propose EditInfinity, which adapts \emph{Infinity}, a binary-quantized generative model, for image editing. We propose an efficient yet effective image inversion mechanism that integrates text prompting rectification and image style preservation, enabling precise image inversion. Furthermore, we devise a holistic smoothing strategy which allows our EditInfinity to perform image editing with high fidelity to source images and precise semantic alignment to the text prompts. Extensive experiments on the PIE-Bench benchmark across `add', `change', and `delete' editing operations, demonstrate the superior performance of our model compared to state-of-the-art diffusion-based baselines. Code available at: https://github.com/yx-chen-ust/EditInfinity.

Figures (13)

• Figure 1: Our method, EditInfinity, delivers strong performance in background preservation in unedited regions and text alignment in edited regions across diverse editing tasks, including add, change and delete object, showing clear advantages over the latest state-of-the-art diffusion-based method RF-Edit wang2024taming, as illustrated by representative examples.
• Figure 2: Image Inversion with Exact Supervision. Given a source image $I_{sou}$ and its prompt $t_{sou}$, we first quantize $I_{sou}$ into exact tokens $R_{1...K}^{sou}$. Then, we concatenate $t_{sou}$ with an instruction $t_{ins}$ and a learnable prompt $t_l$, which is optimized via $\mathcal{L}_{inv}$ under the supervision of $R_{1...K}^{sou}$. Afterwards, the prompt is frozen, and LoRA is applied to the FFN layers of Infinity to further reconstruct $I_{sou}$.
• Figure 3: Image Editing with Holistic Smoothing. First, source image is encoded into $R_{1...K}^{sou}$. At each step $k$ of autoregressive generation, generated $R_k^{tar}$ is conditioned on the concatenation of the target prompt $t_{tar}$, instruction $t_{ins}$, and optimized learnable prompt $t_l$ and then, is blended with $R_k^{sou}$ guided by piecewise linear smoothing kernel $\mathcal{G}$, forming edited tokens $E_k^{tar}$ to prepare for guiding the next-scale generation. Finally, $E_{1...K}^{tar}$ is decoded into the edited image.
• Figure 4: $\mathcal{G}$ as a function of $d$, enabling smooth transitions from edited to unedited regions.
• Figure 5: Comprehensive performance evaluation on PIE-Bench. (a) and (b) report background preservation and text alignment metrics across nine tasks. (c) summarizes user study preferences.