Diffusion-Based Conditional Image Editing through Optimized Inference with Guidance

TL;DR

The paper tackles text-driven image-to-image translation with diffusion models by eliminating fine-tuning and introducing a training-free Optimized Inference with Guidance (OIG) framework. OIG augments the diffusion model's reverse process with a representation-guidance loss $L^{\text{dist}}_t$ that combines CLIP-based semantic alignment to the target prompt with a structural constraint derived from intermediate diffusion features, formulated as a triplet-based objective. Through seconds-level optimization of the target latent trajectory during sampling, OIG achieves improved fidelity to the target description while preserving the source image's structure and background. Empirical results on LAION-5B-based tasks against several training-free baselines show strong quantitative and qualitative performance, with ablations confirming the efficacy of the triplet-based guidance and its robustness across settings.

Abstract

We present a simple but effective training-free approach for text-driven image-to-image translation based on a pretrained text-to-image diffusion model. Our goal is to generate an image that aligns with the target task while preserving the structure and background of a source image. To this end, we derive the representation guidance with a combination of two objectives: maximizing the similarity to the target prompt based on the CLIP score and minimizing the structural distance to the source latent variable. This guidance improves the fidelity of the generated target image to the given target prompt while maintaining the structure integrity of the source image. To incorporate the representation guidance component, we optimize the target latent variable of diffusion model's reverse process with the guidance. Experimental results demonstrate that our method achieves outstanding image-to-image translation performance on various tasks when combined with the pretrained Stable Diffusion model.

Figures (33)

• Figure 1: Overview of the proposed method about utilizing the representation guidance.
• Figure 2: Qualitative comparisons between the proposed algorithm and state-of-the-art methods couairon2022diffedittumanyan2023plugparmar2023zeromokady2023nullcao2023masactrlbrooks2023instructpix2pix on the data sampled from the LAION-5B dataset schuhmann2022laion using the pretrained Stable Diffusion rombach2022high. Note that we strictly keep the original aspect ratio of each image during all experiments, and we change the aspect ratio only for visualization.
• Figure 3: Qualitative comparisons between the proposed algorithm and Pix2Pix-Zero parmar2023zero on synthetic images given by Stable Diffusion rombach2022high.
• Figure 4: Editing examples of the proposed method on synthetic images given by the pretrained Stable Diffusion rombach2022high. Given the source and target prompts, our method generates a target image while successfully preserving the overall structure of the source image and maintaining the background excluding the parts to be manipulated.
• Figure 5: Qualitative reseults of the sensitivity analysis about the hyperparameter $\lambda_2$ using the pretrained Stable Diffusion rombach2022high and real images sampled from the LAION-5B dataset schuhmann2022laion.