Few-Step Diffusion via Score identity Distillation

TL;DR

This work introduces a Diffusion GAN-based adversarial loss applied to the uniform mixture and proposes two new guidance strategies: Zero-CFG, which disables CFG in the teacher and removes text conditioning in the fake score network, and Anti-CFG, which applies negative CFG in the fake score network.

Abstract

Diffusion distillation has emerged as a promising strategy for accelerating text-to-image (T2I) diffusion models by distilling a pretrained score network into a one- or few-step generator. While existing methods have made notable progress, they often rely on real or teacher-synthesized images to perform well when distilling high-resolution T2I diffusion models such as Stable Diffusion XL (SDXL), and their use of classifier-free guidance (CFG) introduces a persistent trade-off between text-image alignment and generation diversity. We address these challenges by optimizing Score identity Distillation (SiD) -- a data-free, one-step distillation framework -- for few-step generation. Backed by theoretical analysis that justifies matching a uniform mixture of outputs from all generation steps to the data distribution, our few-step distillation algorithm avoids step-specific networks and integrates seamlessly into existing pipelines, achieving state-of-the-art performance on SDXL at 1024x1024 resolution. To mitigate the alignment-diversity trade-off when real text-image pairs are available, we introduce a Diffusion GAN-based adversarial loss applied to the uniform mixture and propose two new guidance strategies: Zero-CFG, which disables CFG in the teacher and removes text conditioning in the fake score network, and Anti-CFG, which applies negative CFG in the fake score network. This flexible setup improves diversity without sacrificing alignment. Comprehensive experiments on SD1.5 and SDXL demonstrate state-of-the-art performance in both one-step and few-step generation settings, along with robustness to the absence of real images. Our efficient PyTorch implementation, along with the resulting one- and few-step distilled generators, will be released publicly as a separate branch at https://github.com/mingyuanzhou/SiD-LSG.

Figures (9)

• Figure 1: Example four-step generations at 1024×1024 using our SiD-based multistep distillation method.
• Figure 2: Left: SD1.5, Right: SDXL. Comparison of four-step SiD-LSG (data-free) and SiD$_2^a$ models trained with three different guidance strategies. All models use Uniform-Step Matching with four generation steps. SiD-LSG shows no clear conflict between decreasing FID and increasing CLIP. SiD$_2^a$ models, initialized from SiD-LSG and enhanced with real data, exhibit continued FID reduction during training, with CLIP scores peaking early and gradually declining.
• Figure 3: Example four-step generations at 1024$\times$1024 resolution using our SiD-based multistep distillation method (SiD with LSG). Note that SiD is a data-free distillation method that does not require access to real images.
• Figure 4: Example four-step generations at 1024$\times$1024 resolution using our SiD-based multistep distillation method (SiD$_2^{\alpha}$ with Zero-CFG). Note that SiD$_2^{\alpha}$ initializes its generator from SiD, which is data-free, and continues training with access to a limited number of real text-image pairs.
• Figure 5: Example four-step generations at 1024$\times$1024 resolution using our SiD-based multistep distillation method (SiD$_2^{\alpha}$ with Anti-CFG). Note that SiD$_2^{\alpha}$ initializes its generator from SiD, which is data-free, and continues training with access to a limited number of real text-image pairs.

Theorems & Definitions (2)

• Lemma 1
• proof : Proof of Lemma \ref{['lemma:dropk']}