Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Step

TL;DR

This work addresses the bottleneck of data-free score distillation by integrating an adversarial loss into Score Identity Distillation (SiD). By reusing the fake-score network’s encoder as a 2D discriminator and aggregating batch-level fakeness on each GPU, SiDA jointly distills a one-step generator while aligning it with real-image signals, without adding new parameters. Empirically, SiDA and its SiD-initialized variant SiD^2A achieve faster convergence and state-of-the-art or surpass the teacher diffusion models across CIFAR-10, ImageNet 64x64, FFHQ, AFHQ-v2, and EDM2 family scales, with notably low FID scores and no classifier-free guidance. The results demonstrate significant practical gains for rapid, high-quality one-step diffusion distillation and point to future enhancements via guidance strategies and broader tasks, with code released for reproducibility.

Abstract

Score identity Distillation (SiD) is a data-free method that has achieved SOTA performance in image generation by leveraging only a pretrained diffusion model, without requiring any training data. However, its ultimate performance is constrained by how accurate the pretrained model captures the true data scores at different stages of the diffusion process. In this paper, we introduce SiDA (SiD with Adversarial Loss), which not only enhances generation quality but also improves distillation efficiency by incorporating real images and adversarial loss. SiDA utilizes the encoder from the generator's score network as a discriminator, allowing it to distinguish between real images and those generated by SiD. The adversarial loss is batch-normalized within each GPU and then combined with the original SiD loss. This integration effectively incorporates the average "fakeness" per GPU batch into the pixel-based SiD loss, enabling SiDA to distill a single-step generator. SiDA converges significantly faster than its predecessor when distilled from scratch, and swiftly improves upon the original model's performance during fine-tuning from a pre-distilled SiD generator. This one-step adversarial distillation method establishes new benchmarks in generation performance when distilling EDM diffusion models, achieving FID scores of 1.110 on ImageNet 64x64. When distilling EDM2 models trained on ImageNet 512x512, our SiDA method surpasses even the largest teacher model, EDM2-XXL, which achieved an FID of 1.81 using classifier-free guidance (CFG) and 63 generation steps. In contrast, SiDA achieves FID scores of 2.156 for size XS, 1.669 for S, 1.488 for M, 1.413 for L, 1.379 for XL, and 1.366 for XXL, all without CFG and in a single generation step. These results highlight substantial improvements across all model sizes. Our code is available at https://github.com/mingyuanzhou/SiD/tree/sida.

Figures (39)

• Figure 1: Illustration of the SiDA algorithm. The generator loss, as defined defined in \ref{['eq:obj-theta-sida']}, is determined by fake images $\boldsymbol{x}_g$, true-score-net-denoised fake images $f_{\phi}(\boldsymbol{x}_t, t)$, fake-score-net-denoised fake images $f_{\psi}(\boldsymbol{x}_t, t)$, and the fake image logits. Meanwhile, the fake score network loss, as defined in \ref{['eq:obj-psi-sida']}, is determined by $f_{\psi}(\boldsymbol{x}_t, t)$ and the fake-score-net encoded logits of both fake and real images.
• Figure 2: Ablation Study of $\alpha$ on distilling an EDM model pretrained on CIFAR-10 (unconditional): Each plot illustrates the relation between the performance, measured by FID (left) and Inception Score (right) vs. the number of training iterations during distillation, across varying values of $\alpha$. The batch sizer is 256. The study underscores the impact of $\alpha$ on both training efficiency and generative fidelity, leading us to select $\alpha = 1.0$ for subsequent experiments.
• Figure 4: Evolution of FIDs for the SiD and SiDA generator during the distillation of the EDM teacher model pretrained on CIFAR-10 unconditional (left) and conditional (middle), with a batch size of 256, and on ImageNet 64x64 (right) with a batch size of 8192, using $\alpha = 1.0$ or $\alpha = 1.2$. The performance of EDM (35 NFEs), along with DMD and Diff-Instruct, is depicted with horizontal lines in purple, green, and red, respectively.
• Figure 6: Analogous plot to \ref{['fig:convergence_speed']} for FFHQ-64x64 with batch size of 512 (left) and AFHQ-v2-64x64 with batch size of 512 (right).
• Figure 7: Evolution of FID scores for the SiDA generator during the distillation of the EDM2-XS teacher model, pretrained on ImageNet 512×512. Training utilized either 5% or 100% of the real training images, with a batch size of 2048 and $\alpha = 1.0$. Horizontal lines indicate the performance of EDM2 without classifier-free guidance (CFG) in purple and with CFG (using 63 NFEs) in red.