Direct Inversion: Optimization-Free Text-Driven Real Image Editing with Diffusion Models

TL;DR

Direct Inversion tackles real-image editing with text prompts without optimization by encoding the input image into the diffusion latent noise space and deterministically reversing to an edited image via $\tilde{x_T}$ and $x_{t-1}$ steps guided by the prompt. It enables broad edits—pose, scene, background, style, color, and even racial attributes—without retraining, and provides tunable hyperparameters to balance edit strength against fidelity. Comprehensive ablations show how inversion/inference steps and text guidance versus noise scales shape the editability–fidelity tradeoff, offering practical guidance for parameter selection. The approach generalizes to any diffusion model, offering a fast, optimization-free pathway for real-image editing with diffusion models while highlighting ethical considerations around identity-related edits.

Abstract

With the rise of large, publicly-available text-to-image diffusion models, text-guided real image editing has garnered much research attention recently. Existing methods tend to either rely on some form of per-instance or per-task fine-tuning and optimization, require multiple novel views, or they inherently entangle preservation of real image identity, semantic coherence, and faithfulness to text guidance. In this paper, we propose an optimization-free and zero fine-tuning framework that applies complex and non-rigid edits to a single real image via a text prompt, avoiding all the pitfalls described above. Using widely-available generic pre-trained text-to-image diffusion models, we demonstrate the ability to modulate pose, scene, background, style, color, and even racial identity in an extremely flexible manner through a single target text detailing the desired edit. Furthermore, our method, which we name $\textit{Direct Inversion}$, proposes multiple intuitively configurable hyperparameters to allow for a wide range of types and extents of real image edits. We prove our method's efficacy in producing high-quality, diverse, semantically coherent, and faithful real image edits through applying it on a variety of inputs for a multitude of tasks. We also formalize our method in well-established theory, detail future experiments for further improvement, and compare against state-of-the-art attempts.

Figures (10)

• Figure 1: Direct Inversion - Real image editing with no optimization or fine-tuning. We demonstrate our method's ability to modulate pose, scene, background, style, color, and racial identity in diverse contexts in a zero-shot manner. In an effort to improve diversity of representation, we illustrate Race/Skin Tone variation with later discussion about best practices to avoid exacerbating existing biases.
• Figure 2: Text-Guided Controlled Style Modulation. Editing a real image using our method, Direct Inversion, to modulate a subject's style and attributes.
• Figure 3: Text-Guided Controlled Item Variation. Editing a real image using our method, Direct Inversion, to modulate an item's style/attributes.
• Figure 4: Direct Inversion Process Diagram.[1] Inversion: First, we encode the input real image into its' encoded noises (respective to timesteps). [2] Decoding/Edit: Then, we take the final inverted noise, pass it into the noise-prediction UNet, merge the outputted noise with the corresponding timestep's inverted noise, and then use that noise to sample the previous timestep. We repeat this process until we reach timestep 0, which corresponds to the resultant edited image.
• Figure 5: DDIM Reconstruction from Inverted Latent Space. As the number of inversion and inference steps increases, the DDIM reconstruction of the original real image becomes better.