Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visual Chain-of-Thought Diffusion Models

William Harvey, Frank Wood

TL;DR

The paper identifies a persistent gap where conditional diffusion models outperform unconditional ones, especially as conditioning information increases. It introduces Visual Chain-of-Thought Diffusion Models (VCDM), a two-stage approach that first samples a semantically rich CLIP embedding and then generates an image conditioned on that embedding before discarding the embedding. Empirically, VCDM achieves substantial improvements in FID over standard unconditional generation across AFHQ, FFHQ, and ImageNet, with modest inference overhead. This method enables leveraging strong conditional DGMs for unconditional or lightly-conditional generation, broadening practical diffusion-model applicability.

Abstract

Recent progress with conditional image diffusion models has been stunning, and this holds true whether we are speaking about models conditioned on a text description, a scene layout, or a sketch. Unconditional image diffusion models are also improving but lag behind, as do diffusion models which are conditioned on lower-dimensional features like class labels. We propose to close the gap between conditional and unconditional models using a two-stage sampling procedure. In the first stage we sample an embedding describing the semantic content of the image. In the second stage we sample the image conditioned on this embedding and then discard the embedding. Doing so lets us leverage the power of conditional diffusion models on the unconditional generation task, which we show improves FID by 25-50% compared to standard unconditional generation.

Visual Chain-of-Thought Diffusion Models

TL;DR

The paper identifies a persistent gap where conditional diffusion models outperform unconditional ones, especially as conditioning information increases. It introduces Visual Chain-of-Thought Diffusion Models (VCDM), a two-stage approach that first samples a semantically rich CLIP embedding and then generates an image conditioned on that embedding before discarding the embedding. Empirically, VCDM achieves substantial improvements in FID over standard unconditional generation across AFHQ, FFHQ, and ImageNet, with modest inference overhead. This method enables leveraging strong conditional DGMs for unconditional or lightly-conditional generation, broadening practical diffusion-model applicability.

Abstract

Recent progress with conditional image diffusion models has been stunning, and this holds true whether we are speaking about models conditioned on a text description, a scene layout, or a sketch. Unconditional image diffusion models are also improving but lag behind, as do diffusion models which are conditioned on lower-dimensional features like class labels. We propose to close the gap between conditional and unconditional models using a two-stage sampling procedure. In the first stage we sample an embedding describing the semantic content of the image. In the second stage we sample the image conditioned on this embedding and then discard the embedding. Doing so lets us leverage the power of conditional diffusion models on the unconditional generation task, which we show improves FID by 25-50% compared to standard unconditional generation.
Paper Structure (14 sections, 2 equations, 4 figures, 1 table)

This paper contains 14 sections, 2 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Conditional DGMs
  4. CLIP embeddings
  5. Conditional vs. unconditional DGMs
  6. What does it mean to say that conditional DGMs beat unconditional DGMs?
  7. Why do conditional DGMs beat unconditional DGMs?
  8. When do conditional DGMs beat unconditional DGMS?
  9. Method
  10. Auxiliary model
  11. Conditional image model
  12. Experiments
  13. Related work
  14. Discussion and conclusion

Figures (4)

  • Figure 1: Left: Output from Stable Diffusion rombach2022high prompted to produce "aerial photography". Right: Using a more detailed promptwith the same random seed removes the "smudged" road artifact that appears on the left. VCDM builds on this observation.
  • Figure 2: CLIP-conditional samples on AFHQ and FFHQ. Each row shows three samples conditioned on the same CLIP embedding.
  • Figure 3: FID versus dimensionality of ${\mathbf{y}}$ on AFHQ choi2020stargan and FFHQ karras2018style. With small training budgets (brown line), it is harmful when ${\mathbf{y}}$ is too informative. With larger training budgets (purple line), it is helpful to make ${\mathbf{y}}$ much more high dimensional.
  • Figure 4: FID throughout training. We show results for each method trained from scratch and, on AFHQ and FFHQ, for finetuning a pretrained EDM model (which was trained for the equivalent of 32 GPU days). VCDM quickly outperforms EDM when trained from scratch and quickly improves on the pretrained model when used for finetuning.