Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Invertible Consistency Distillation for Text-Guided Image Editing in Around 7 Steps

Nikita Starodubcev, Mikhail Khoroshikh, Artem Babenko, Dmitry Baranchuk

TL;DR

The paper tackles the limitation of distilled diffusion models lacking reliable image inversion by introducing invertible Consistency Distillation (iCD), a framework combining forward and reverse consistency models with a multi-boundary design to enable deterministic multistep inversion. It couples forward fCD^m and reverse CD^m training with forward and reverse preservation losses and augments the process with dynamic classifier-free guidance to improve inversion without harming generation. Empirical results on SD1.5 and SDXL show that 3-4 steps suffice for high-quality inversion and 6-8 steps for effective editing, with iCD matching or exceeding state-of-the-art baselines while remaining significantly faster. This approach broadens the practical utility of distilled diffusion for zero-shot text-guided image editing and paves the way for real-time applications.

Abstract

Diffusion distillation represents a highly promising direction for achieving faithful text-to-image generation in a few sampling steps. However, despite recent successes, existing distilled models still do not provide the full spectrum of diffusion abilities, such as real image inversion, which enables many precise image manipulation methods. This work aims to enrich distilled text-to-image diffusion models with the ability to effectively encode real images into their latent space. To this end, we introduce invertible Consistency Distillation (iCD), a generalized consistency distillation framework that facilitates both high-quality image synthesis and accurate image encoding in only 3-4 inference steps. Though the inversion problem for text-to-image diffusion models gets exacerbated by high classifier-free guidance scales, we notice that dynamic guidance significantly reduces reconstruction errors without noticeable degradation in generation performance. As a result, we demonstrate that iCD equipped with dynamic guidance may serve as a highly effective tool for zero-shot text-guided image editing, competing with more expensive state-of-the-art alternatives.

Invertible Consistency Distillation for Text-Guided Image Editing in Around 7 Steps

TL;DR

The paper tackles the limitation of distilled diffusion models lacking reliable image inversion by introducing invertible Consistency Distillation (iCD), a framework combining forward and reverse consistency models with a multi-boundary design to enable deterministic multistep inversion. It couples forward fCD^m and reverse CD^m training with forward and reverse preservation losses and augments the process with dynamic classifier-free guidance to improve inversion without harming generation. Empirical results on SD1.5 and SDXL show that 3-4 steps suffice for high-quality inversion and 6-8 steps for effective editing, with iCD matching or exceeding state-of-the-art baselines while remaining significantly faster. This approach broadens the practical utility of distilled diffusion for zero-shot text-guided image editing and paves the way for real-time applications.

Abstract

Diffusion distillation represents a highly promising direction for achieving faithful text-to-image generation in a few sampling steps. However, despite recent successes, existing distilled models still do not provide the full spectrum of diffusion abilities, such as real image inversion, which enables many precise image manipulation methods. This work aims to enrich distilled text-to-image diffusion models with the ability to effectively encode real images into their latent space. To this end, we introduce invertible Consistency Distillation (iCD), a generalized consistency distillation framework that facilitates both high-quality image synthesis and accurate image encoding in only 3-4 inference steps. Though the inversion problem for text-to-image diffusion models gets exacerbated by high classifier-free guidance scales, we notice that dynamic guidance significantly reduces reconstruction errors without noticeable degradation in generation performance. As a result, we demonstrate that iCD equipped with dynamic guidance may serve as a highly effective tool for zero-shot text-guided image editing, competing with more expensive state-of-the-art alternatives.
Paper Structure (20 sections, 5 equations, 22 figures, 6 tables)

This paper contains 20 sections, 5 equations, 22 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Method
  4. Forward Consistency Distillation
  5. Multi-boundary Consistency Distillation
  6. Training fCD$^{m}$ and CD$^{m}$
  7. Dynamic Classifier-Free Guidance Facilitates Inversion
  8. Experiments
  9. Inversion quality of iCD
  10. Text-guided image editing
  11. Text-guided image editing with iCD-SD1.5
  12. Text-guided image editing with iCD-XL
  13. Conclusion
  14. Training details of iCD
  15. Image generation with iCD
  16. ...and 5 more sections

Figures (22)

  • Figure 1: Invertible Consistency Distillation (iCD) enables both fast image editing and strong generation performance in a few model evaluations.
  • Figure 2: Dynamic CFG strategies.
  • Figure 3: The proposed invertible Consistency Distillation framework consists of two models: the forward $m$-boundary model, fCD$^{m}$, and the reverse model, CD$^{m}$. (a) For $m=1$, the reverse model corresponds to CD. More boundary points unlock the deterministic multistep inversion, e.g., (b) shows the case for $m=2$.
  • Figure 4: (a) Reconstruction error of the decoding process for different CFG turn-on thresholds. (b) Image inversion examples for different CFG turn-on thresholds $\mathbf{T}$. Guidance at high noise levels ($\mathbf{T}=1.0$) drastically degrades the inversion quality.
  • Figure 5: (a) Trade-off between generation performance (IR) and reconstruction quality (MSE) provided by different $\tau_{1}, \tau_{2}$. (b) Generation examples for dynamic and constant CFG scales. The points around $\tau_1 = \tau_2 = 0.8$ provide preferable trade-off between generation and inversion performance.
  • ...and 17 more figures