Efficient Distillation of Classifier-Free Guidance using Adapters

TL;DR

This work tackles the computational overhead of classifier-free guidance (CFG) in diffusion models by introducing adapter guidance distillation (AGD). AGD trains lightweight adapters on cfg-guided trajectories while keeping the base diffusion model frozen, effectively embedding CFG into a single forward pass and doubling sampling speed without full-model fine-tuning. The key contributions are training on CFG-guided trajectories, a residual adapter formulation, and two adapter architectures (cross-attention and offset) that achieve comparable or better FID with far fewer trainable parameters and on consumer hardware. Empirically, AGD matches or surpasses CFG across several architectures (e.g., dit, SD2, SDXL), maintains robustness to unseen guidance scales, and supports combining with other adapters, all while drastically reducing resource requirements during training. This makes efficient guided diffusion more accessible for large models and broad applications, with practical impact on speed, memory usage, and flexibility of model composition.

Abstract

While classifier-free guidance (CFG) is essential for conditional diffusion models, it doubles the number of neural function evaluations (NFEs) per inference step. To mitigate this inefficiency, we introduce adapter guidance distillation (AGD), a novel approach that simulates CFG in a single forward pass. AGD leverages lightweight adapters to approximate CFG, effectively doubling the sampling speed while maintaining or even improving sample quality. Unlike prior guidance distillation methods that tune the entire model, AGD keeps the base model frozen and only trains minimal additional parameters ($\sim$2%) to significantly reduce the resource requirement of the distillation phase. Additionally, this approach preserves the original model weights and enables the adapters to be seamlessly combined with other checkpoints derived from the same base model. We also address a key mismatch between training and inference in existing guidance distillation methods by training on CFG-guided trajectories instead of standard diffusion trajectories. Through extensive experiments, we show that AGD achieves comparable or superior FID to CFG across multiple architectures with only half the NFEs. Notably, our method enables the distillation of large models ($\sim$2.6B parameters) on a single consumer GPU with 24 GB of VRAM, making it more accessible than previous approaches that require multiple high-end GPUs. We will publicly release the implementation of our method.

Figures (21)

• Figure 1: Generated samples using agd applied to various models. By efficiently baking cfg into the base diffusion model, agd generates high-quality samples with only a single forward pass per inference step, which results in doubling the sampling speed compared to standard cfg.
• Figure 2: High-level overview of AGD components. (a) Instead of training on diffusion trajectories, we first run the sampling process with classifier-free guidance (CFG) and use the resulting guided trajectories (i.e., intermediate predictions at each time step $t$) as our training dataset. (b) We then introduce small adapters to the base model and train them to replicate the CFG-guided predictions from (a) while keeping the base model frozen. (c) During inference, the base model combined with the trained adapter produces guided predictions in a single forward pass, effectively doubling the sampling speed compared to CFG.
• Figure 3: One-dimensional illustration of the mismatch between standard diffusion trajectories used for training in existing guidance distillation methods and the actual guided trajectories followed during inference. The cfg trajectories in (c) occupy regions in space distinct from the standard diffusion trajectories in (b). Training directly on cfg-guided trajectories ensures that the adapters focus on the regions primarily encountered during sampling with cfg.
• Figure 4: Visual illustration of the trainable adapters alongside the frozen base model. The adapters are typically integrated with attention layers (either self-attention or cross-attention), and their outputs are added to those of the frozen attention blocks.
• Figure 5: Qualitative comparison between agd and cfg. agd produces samples with comparable quality to cfg while achieving twice the inference speed by requiring only a single forward pass through the model. Additionally, agd samples maintain structural similarity to cfg but often have better visual coherence.