Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Manifold Preserving Guided Diffusion

Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J. Zico Kolter, Ruslan Salakhutdinov, Stefano Ermon

TL;DR

MPGD introduces a training-free, manifold-aware framework for conditional diffusion sampling that constrains guidance to the data manifold via tangent-space updates. It provides a practical shortcut (DDIM-based) and multiple projection routes, including a perfect-autoencoder and latent-space variants, to preserve manifold geometry during sampling. Across pixel-space and latent diffusion models, MPGD delivers up to 3.8x speedups with maintained or improved sample quality on inverse problems, FaceID guidance, and style-guided generation. This approach enables broadly applicable, low-cost conditional generation while highlighting considerations for safety and biases in large pretrained models.

Abstract

Despite the recent advancements, conditional image generation still faces challenges of cost, generalizability, and the need for task-specific training. In this paper, we propose Manifold Preserving Guided Diffusion (MPGD), a training-free conditional generation framework that leverages pretrained diffusion models and off-the-shelf neural networks with minimal additional inference cost for a broad range of tasks. Specifically, we leverage the manifold hypothesis to refine the guided diffusion steps and introduce a shortcut algorithm in the process. We then propose two methods for on-manifold training-free guidance using pre-trained autoencoders and demonstrate that our shortcut inherently preserves the manifolds when applied to latent diffusion models. Our experiments show that MPGD is efficient and effective for solving a variety of conditional generation applications in low-compute settings, and can consistently offer up to 3.8x speed-ups with the same number of diffusion steps while maintaining high sample quality compared to the baselines.

Manifold Preserving Guided Diffusion

TL;DR

MPGD introduces a training-free, manifold-aware framework for conditional diffusion sampling that constrains guidance to the data manifold via tangent-space updates. It provides a practical shortcut (DDIM-based) and multiple projection routes, including a perfect-autoencoder and latent-space variants, to preserve manifold geometry during sampling. Across pixel-space and latent diffusion models, MPGD delivers up to 3.8x speedups with maintained or improved sample quality on inverse problems, FaceID guidance, and style-guided generation. This approach enables broadly applicable, low-cost conditional generation while highlighting considerations for safety and biases in large pretrained models.

Abstract

Despite the recent advancements, conditional image generation still faces challenges of cost, generalizability, and the need for task-specific training. In this paper, we propose Manifold Preserving Guided Diffusion (MPGD), a training-free conditional generation framework that leverages pretrained diffusion models and off-the-shelf neural networks with minimal additional inference cost for a broad range of tasks. Specifically, we leverage the manifold hypothesis to refine the guided diffusion steps and introduce a shortcut algorithm in the process. We then propose two methods for on-manifold training-free guidance using pre-trained autoencoders and demonstrate that our shortcut inherently preserves the manifolds when applied to latent diffusion models. Our experiments show that MPGD is efficient and effective for solving a variety of conditional generation applications in low-compute settings, and can consistently offer up to 3.8x speed-ups with the same number of diffusion steps while maintaining high sample quality compared to the baselines.
Paper Structure (64 sections, 10 theorems, 31 equations, 24 figures, 5 tables, 3 algorithms)

This paper contains 64 sections, 10 theorems, 31 equations, 24 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Conventional Training Free Guided Diffusion
  3. Problem Formulation
  4. Diffusion Models
  5. Training-Free Guided diffusion
  6. Other related works
  7. Issues in the previous formulation: The Manifold Hypothesis
  8. Manifold Preserving Training-Free Guided Diffusion
  9. Objective
  10. Method
  11. The MPGD Shortcut
  12. Manifold Projection with (Perfect) Autoencoders
  13. Manipulating the Latents
  14. MPGD with Latent Diffusion Models
  15. Multi-step Optimization
  16. ...and 49 more sections

Key Result

Proposition 1

Define $d(x,\nu, \mathcal{M}) := \inf_{x' \in \mathcal{M}}\|x-\nu x'\|_2$ for $\nu>0$, and $B(\mathcal{M};r) := \{x \in \mathbb{R}^d \mid d(x,1,\mathcal{M}) < r\}$ for $r>0$. Consider the distribution of noisy data $p_t(x_t):=\int p(x_t|x)p(x)dx$, where $p(x_t|x):=\mathcal{N}(\sqrt{\bar{\alpha}_t}x,

Figures (24)

  • Figure 1: Our proposed MPGD as a training-free sampling method for both pre-trained pixel-space diffusion models and latent diffusion models in a variety of conditional generation applications. MPGD can be applied to a broad range of tasks with minimal sampling time and high sample quality.
  • Figure 2: A schematic overview of our proposed approaches and an illustrative comparison with DDIM ddim and DPS dps.
  • Figure 3: MPGD for pixel diffusion models
  • Figure 4: Qualitative examples of solving noisy linear inverse problems with our proposed MPGD and baseline DPS.
  • Figure 5: Quantitative results of FFHQ super-resolution experiment that compares fidelity (log KID), guidance quality (LPIPS) and inference time across different numbers of DDIM steps.
  • ...and 19 more figures

Theorems & Definitions (20)

  • Proposition 1: Concentration of Noisy Samples (Informal, extended from mcgfastsampling)
  • Theorem 1
  • Theorem 2
  • proof
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • proof
  • Proposition 2
  • ...and 10 more