Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesis

TL;DR

This work tackles mode collapse and limited diversity in diffusion model distillation by introducing Adversarial Distribution Matching (ADM), which uses diffusion-based discriminators to implicitly align real and fake score predictions. The authors propose DMDX, a unified pipeline that combines ADM fine-tuning with adversarial distillation pre-training (ADP), including a cubic generator timestep schedule and dual discriminators in latent and pixel spaces. Empirical results show strong one-step performance on SDXL and new benchmarks for multi-step ADM across SD3 and CogVideoX, achieving substantial speedups with competitive fidelity and improved diversity. The approach offers a flexible, data-driven discrepancy measure that mitigates the limitations of predefined divergences and demonstrates practical impact for efficient image and video synthesis.

Abstract

Distribution Matching Distillation (DMD) is a promising score distillation technique that compresses pre-trained teacher diffusion models into efficient one-step or multi-step student generators. Nevertheless, its reliance on the reverse Kullback-Leibler (KL) divergence minimization potentially induces mode collapse (or mode-seeking) in certain applications. To circumvent this inherent drawback, we propose Adversarial Distribution Matching (ADM), a novel framework that leverages diffusion-based discriminators to align the latent predictions between real and fake score estimators for score distillation in an adversarial manner. In the context of extremely challenging one-step distillation, we further improve the pre-trained generator by adversarial distillation with hybrid discriminators in both latent and pixel spaces. Different from the mean squared error used in DMD2 pre-training, our method incorporates the distributional loss on ODE pairs collected from the teacher model, and thus providing a better initialization for score distillation fine-tuning in the next stage. By combining the adversarial distillation pre-training with ADM fine-tuning into a unified pipeline termed DMDX, our proposed method achieves superior one-step performance on SDXL compared to DMD2 while consuming less GPU time. Additional experiments that apply multi-step ADM distillation on SD3-Medium, SD3.5-Large, and CogVideoX set a new benchmark towards efficient image and video synthesis.

Figures (16)

• Figure 1: In these images, some are generated by the baseline SDXL via 50NFE, the others with our DMDX in 1NFE. Can you tell which is the accelerated one? Answers in the footnote\ref{['footnote3']}.
• Figure 2: Overall pipeline of our proposed Adversarial Distribution Matching (ADM) and Adversarial Distillation Pre-training (ADP).
• Figure 3: Changes of DMD loss over multi-step ADM distillation for CogVideoX. Note that we did not optimize this objective directly during ADM distillation but recorded it over iterations.
• Figure 4: Illustration for theoretical discussion.
• Figure 5: Qualitative results on fully fine-tuning SDXL-Base.