ADT: Tuning Diffusion Models with Adversarial Supervision

TL;DR

This work tackles training-inference divergence in diffusion models by introducing Adversarial Diffusion Tuning (ADT), which simulates the inference process during optimization and uses adversarial supervision to align outputs with training data. ADT employs a siamese-discriminator with a frozen backbone (e.g., DINOv2) and lightweight heads, plus an image-to-image inference path and differentiable data augmentation to stabilize training while preserving the original diffusion loss ${\mathcal L}_{diff}$. Gradient flow along the iterative inference path is managed via a backward-flow constraint that stops gradients on inputs and updates only a subset of steps, preventing memory blowups and gradient explosions. Empirical results on Stable Diffusion variants (SD15, SDXL, SD3) with a 170k JourneyDB training set show improved distribution alignment (lower FID) and higher image quality (HPS/AES), with robust generalization across samplers and CFG settings and informative ablations supporting the design choices.

Abstract

Diffusion models have achieved outstanding image generation by reversing a forward noising process to approximate true data distributions. During training, these models predict diffusion scores from noised versions of true samples in a single forward pass, while inference requires iterative denoising starting from white noise. This training-inference divergences hinder the alignment between inference and training data distributions, due to potential prediction biases and cumulative error accumulation. To address this problem, we propose an intuitive but effective fine-tuning framework, called Adversarial Diffusion Tuning (ADT), by stimulating the inference process during optimization and aligning the final outputs with training data by adversarial supervision. Specifically, to achieve robust adversarial training, ADT features a siamese-network discriminator with a fixed pre-trained backbone and lightweight trainable parameters, incorporates an image-to-image sampling strategy to smooth discriminative difficulties, and preserves the original diffusion loss to prevent discriminator hacking. In addition, we carefully constrain the backward-flowing path for back-propagating gradients along the inference path without incurring memory overload or gradient explosion. Finally, extensive experiments on Stable Diffusion models (v1.5, XL, and v3), demonstrate that ADT significantly improves both distribution alignment and image quality.

Figures (15)

• Figure 1: A simple illustration for the training-inference divergence. The training procedure aims to fit the score function $\nabla_x \log p(x,t)$ at every time step with access to real data, while the inference turns to denoise from white noise through a fixed path without the exact score estimation and real data.
• Figure 2: The overview framework of our ADT model.
• Figure 3: The win rate of different training strategies ranked by quality assessment models (Top) and human experts (Bottom).
• Figure 4: The generated samples under different training strategies, where captions sampled from HPSV2 Benchmark prompts.
• Figure 5: The performance of ADT with different CFG scales (Top) or numbers of inference steps (Bottom).