REED-VAE: RE-Encode Decode Training for Iterative Image Editing with Diffusion Models

TL;DR

This work tackles the challenge of artifact accumulation during multi-method iterative image editing with diffusion models by introducing REED-VAE, a VAE whose decoder is finetuned to preserve fidelity across repeated encode–decode cycles. It achieves this through an iterative training regime with dynamic curriculum (varying k), a first-step loss, and a combined objective that includes MSE, perceptual similarity, and latent regularization. REED-VAE can be swapped into existing SD-based editing pipelines to allow seamless mixing of pixel-space and latent-space edits while maintaining high image quality over many iterations, as demonstrated across text- and mask-guided and exemplar-driven edits. The approach shows consistent improvements over vanilla VAEs and outperforms inversion-based methods in iterative scenarios, highlighting its potential as a versatile benchmark and component for robust, multi-method image editing in practical workflows.

Abstract

While latent diffusion models achieve impressive image editing results, their application to iterative editing of the same image is severely restricted. When trying to apply consecutive edit operations using current models, they accumulate artifacts and noise due to repeated transitions between pixel and latent spaces. Some methods have attempted to address this limitation by performing the entire edit chain within the latent space, sacrificing flexibility by supporting only a limited, predetermined set of diffusion editing operations. We present a RE-encode decode (REED) training scheme for variational autoencoders (VAEs), which promotes image quality preservation even after many iterations. Our work enables multi-method iterative image editing: users can perform a variety of iterative edit operations, with each operation building on the output of the previous one using both diffusion-based operations and conventional editing techniques. We demonstrate the advantage of REED-VAE across a range of image editing scenarios, including text-based and mask-based editing frameworks. In addition, we show how REED-VAE enhances the overall editability of images, increasing the likelihood of successful and precise edit operations. We hope that this work will serve as a benchmark for the newly introduced task of multi-method image editing. Our code and models will be available at https://github.com/galmog/REED-VAE

Figures (10)

• Figure 1: Even without a diffusion model in the pipeline, the Vanilla-VAE (top row) accumulates artifacts and exhibits significant distortion very quickly throughout encode-decode iterations. The tiger's features lose their distinct shapes and edges, appearing more globular and less defined. The color palette is altered, with a noticeable increase in blue tones and a decrease in the richness of the orange and greens. Fine details such as the grass and the fur are largely lost or blurred. REED-VAE (bottom row), produces successive images that are robust to such artifacts and distortions. The tiger retains its shape, color, and surface details, demonstrating remarkably high fidelity to the original image. The subtle variations in orange and white hues are preserved, and fine elements remain visible.
• Figure 2: Given an input image (a) we perform 20 encode-decode iterations and present the the results in image (top) and frequency domain (bottom). Vanilla-VAE (b) exhibits significant loss of high-frequency information (evidenced by the dimming and blurring of the outer regions of the spectrum), and dominance of low-frequency features (evidenced by the enlarged central bright region). In addition, it also introduced new high-frequency features that are not seen in the input image, indicating an introduction of repetitive artifacts. Trying to apply smoothing after each encode-decode iteration (c) solves some of these problems at the cost of blurring the image. REED-VAE (d) demonstrates superior performance in preserving image fidelity across all frequency bands.
• Figure 3: Examples of types of edit sessions made possible with REED-VAE. Using the Vanilla-VAE (right), significant noise and artifacts accumulate quickly after multiple edit operations. Intermediate edit operations are omitted to highlight the final edited image. Four types of edit operations are performed: text-guided editing brooks2023instructpix2pix, external editing (not diffusion-based), mask-guided editing avrahami2023blended, and example-guided editing yang2023paint.
• Figure 4: Row 1: Null-Text Inversion (NTI) is used to iteratively invert the image and regenerate it from the inverted latent. Row 2: the Vanilla-VAE is used to iteratively encode and decode the image. Row 3: NTI is used with REED-VAE to iteratively invert the image and regenerate it from the inverted latent. Vanilla NTI loses fidelity to the original image and is not resilient to iterative degradation. Full sequences for Vanilla NTI and NTI + REED are available in the Supplementary Material.
• Figure 5: Iterative edits using Null-Text Inversion. Prompts: "a landscape with desert mountains" $\to$ "a landscape with snowy mountains". Despite regenerating latents through the inversion process, visual artifacts accumulate, particularly in later iterations (e.g. noise patterns and loss of fidelity to the original image). This illustrates that DDIM inversion-based methods do not mitigate the degradation that occurs in iterative editing tasks, underscoring the need for REED-VAE.