Beyond and Free from Diffusion: Invertible Guided Consistency Training

TL;DR

The paper addresses saturation artifacts and compute bottlenecks in guidance for diffusion-based generative models by proposing invertible Guided Consistency Training (iGCT), a DM-independent, data-driven framework for guided consistency models. iGCT integrates guided consistency training with inverse consistency training and introduces a noiser component to map images to a noise latent, enabling invertible, one-stage training that supports fast, 1-step guided generation and editing. Empirical results on CIFAR-10 and ImageNet64 show improved FID and precision under high guidance compared to classifier-free guidance and distillation-based consistency models, along with rapid inversion-based editing. This approach reduces training complexity and computational cost while delivering robust guidance and inversion capabilities, though it currently exhibits some limitations on ImageNet64 and benefits from stronger theoretical grounding.

Abstract

Guidance in image generation steers models towards higher-quality or more targeted outputs, typically achieved in Diffusion Models (DMs) via Classifier-free Guidance (CFG). However, recent Consistency Models (CMs), which offer fewer function evaluations, rely on distilling CFG knowledge from pretrained DMs to achieve guidance, making them costly and inflexible. In this work, we propose invertible Guided Consistency Training (iGCT), a novel training framework for guided CMs that is entirely data-driven. iGCT, as a pioneering work, contributes to fast and guided image generation and editing without requiring the training and distillation of DMs, greatly reducing the overall compute requirements. iGCT addresses the saturation artifacts seen in CFG under high guidance scales. Our extensive experiments on CIFAR-10 and ImageNet64 show that iGCT significantly improves FID and precision compared to CFG. At a guidance of 13, iGCT improves precision to 0.8, while DM's drops to 0.47. Our work takes the first step toward enabling guidance and inversion for CMs without relying on DMs.

Figures (31)

• Figure 1: Guidance of EDM trained via Classifier-free Guidance (CFG) (top-left), and our Invertible Guided Consistency Training (iGCT) (bottom-left). iGCT enables fast inverse-based image editing while preserving the source semantics (right). Unlike CFG, iGCT eliminates the need for two-stage training and removes contrast artifacts, achieving better precision and FID under high guidance (Fig. \ref{['fig:results_fid_prec_rec']}).
• Figure 2: Timeline of key developments in diffusion training/sampling schedulers (top). The schedulers used by pretrained DMs that CD depends on (bottom). The fields of DM and CT are advancing rapidly in multiple directions, and tightly coupling CM with DM would slow down progress. To train a CM under a particular diffusion scheduler, CD-based CMs rely on a teacher DM trained under the same setting, while CT is free of such constraint.
• Figure 3: Dynamics of guidance, CFG vs iGCT, 1D toy example with 2 modes at $x=2$ and $-2$. Under high guidance, CFG overshoots (in orange), causing the modes to fall off and producing high contrast values. Our iGCT (in green) is able to preserve the modes same as the data distribution (in blue).
• Figure 4: Overview of iGCT. As oppose to the denoiser, the noiser learns to map image to noise by swapping the model's input at training, i.e., the noisier sample $\boldsymbol{x}_t$ is $\boldsymbol{x}_r$'s target (See Eq. \ref{['eq:inv-loss']} for details). The predicted noise latent and denoised image distribution is aligned using the reconstruction loss.
• Figure 5: In traditional conditional CT, estimating the unconditional noise $\epsilon_{\emptyset}$ from a noisy sample $\boldsymbol{x}_t$ requires averaging over all potential clean images (left). By decoupling the target class $c^\text{tar}$ from the source image $\boldsymbol{x}_0^{\text{src}}$, we can estimate $\epsilon_{\emptyset}$ with $z$ as how unconditional CT predicts noise, and $\epsilon_{\text{c}}^{\text{tar}}$ with $z^*$ like how conditional CT predicts noise (right).