Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models

Tariq Berrada Ifriqi, Pietro Astolfi, Melissa Hall, Reyhane Askari-Hemmat, Yohann Benchetrit, Marton Havasi, Matthew Muckley, Karteek Alahari, Adriana Romero-Soriano, Jakob Verbeek, Michal Drozdzal

TL;DR

This work addresses reproducibility and efficiency in large-scale latent diffusion models by systematically evaluating conditioning mechanisms and pre-training strategies. It introduces a disentangled conditioning framework that separates semantic prompts from control metadata, enhances text conditioning with noisy replicate padding, and integrates classifier-free guidance to independently tune semantic and control influences. It also investigates transfer learning across datasets and resolutions, proposing positional-embedding resampling and noise-schedule scaling to maintain performance during scale-up, with cropping strategies tailored to resolution. Empirically, the approach achieves state-of-the-art FID on ImageNet-1k at 512 and CC12M at 512 (e.g., FID 2.76 and 8.64 respectively, with CLIP scores improving to 26.17), while reducing training costs and improving transferability. These results offer practical, reproducible guidelines for conditioning and pre-training diffusion models at scale, alongside a discussion of limitations and societal impact.

Abstract

Large-scale training of latent diffusion models (LDMs) has enabled unprecedented quality in image generation. However, the key components of the best performing LDM training recipes are oftentimes not available to the research community, preventing apple-to-apple comparisons and hindering the validation of progress in the field. In this work, we perform an in-depth study of LDM training recipes focusing on the performance of models and their training efficiency. To ensure apple-to-apple comparisons, we re-implement five previously published models with their corresponding recipes. Through our study, we explore the effects of (i)~the mechanisms used to condition the generative model on semantic information (e.g., text prompt) and control metadata (e.g., crop size, random flip flag, etc.) on the model performance, and (ii)~the transfer of the representations learned on smaller and lower-resolution datasets to larger ones on the training efficiency and model performance. We then propose a novel conditioning mechanism that disentangles semantic and control metadata conditionings and sets a new state-of-the-art in class-conditional generation on the ImageNet-1k dataset -- with FID improvements of 7% on 256 and 8% on 512 resolutions -- as well as text-to-image generation on the CC12M dataset -- with FID improvements of 8% on 256 and 23% on 512 resolution.

On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models

TL;DR

This work addresses reproducibility and efficiency in large-scale latent diffusion models by systematically evaluating conditioning mechanisms and pre-training strategies. It introduces a disentangled conditioning framework that separates semantic prompts from control metadata, enhances text conditioning with noisy replicate padding, and integrates classifier-free guidance to independently tune semantic and control influences. It also investigates transfer learning across datasets and resolutions, proposing positional-embedding resampling and noise-schedule scaling to maintain performance during scale-up, with cropping strategies tailored to resolution. Empirically, the approach achieves state-of-the-art FID on ImageNet-1k at 512 and CC12M at 512 (e.g., FID 2.76 and 8.64 respectively, with CLIP scores improving to 26.17), while reducing training costs and improving transferability. These results offer practical, reproducible guidelines for conditioning and pre-training diffusion models at scale, alongside a discussion of limitations and societal impact.

Abstract

Large-scale training of latent diffusion models (LDMs) has enabled unprecedented quality in image generation. However, the key components of the best performing LDM training recipes are oftentimes not available to the research community, preventing apple-to-apple comparisons and hindering the validation of progress in the field. In this work, we perform an in-depth study of LDM training recipes focusing on the performance of models and their training efficiency. To ensure apple-to-apple comparisons, we re-implement five previously published models with their corresponding recipes. Through our study, we explore the effects of (i)~the mechanisms used to condition the generative model on semantic information (e.g., text prompt) and control metadata (e.g., crop size, random flip flag, etc.) on the model performance, and (ii)~the transfer of the representations learned on smaller and lower-resolution datasets to larger ones on the training efficiency and model performance. We then propose a novel conditioning mechanism that disentangles semantic and control metadata conditionings and sets a new state-of-the-art in class-conditional generation on the ImageNet-1k dataset -- with FID improvements of 7% on 256 and 8% on 512 resolutions -- as well as text-to-image generation on the CC12M dataset -- with FID improvements of 8% on 256 and 23% on 512 resolution.

Paper Structure

This paper contains 31 sections, 2 theorems, 17 equations, 11 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Conditioning and pre-training strategies for diffusion models
  3. Conditioning mechanisms
  4. On transferring models pre-trained on different datasets and resolutions
  5. Experimental evaluation
  6. Experimental setup
  7. Evaluation of model architectures and comparison with the state of the art
  8. Control conditioning
  9. Transferring weights between datasets and resolutions
  10. Conclusion
  11. Appendix
  12. Related work
  13. Societal impact and limitations
  14. Implementation details
  15. Noise schedule scaling.
  16. ...and 16 more sections

Key Result

Proposition 1

When going from a scale of $s$ to a scale $s'$, we update the $\beta$ scheduler according to the following rule

Figures (11)

  • Figure 1: Qualitative examples. Images generated using our model trained on CC12M at 512 resolution.
  • Figure 2: Weighting of low-level control conditions. The weight is zeroed out early on when image semantics are defined, and increased later when adding details.
  • Figure 3: Interpolation and extrapolation of positional embeddings.
  • Figure 4: Low-resolution pre-training. Crop size used for pre-training impacts finetuning.
  • Figure 5: Illustration of the impact of flip conditioning. Without the flip conditioning, the model may confuse left-right specifications. Including flipping as a control condition enables the model to properly follow left-right instructions.
  • ...and 6 more figures

Theorems & Definitions (5)

  • Proposition 1
  • Proposition 2
  • proof
  • proof
  • proof