Decoupling Training-Free Guided Diffusion by ADMM

TL;DR

This paper proposes a novel framework that distinctly decouples the unconditional diffusion model and the guided loss through a tuned weight hyperparameter, and develops a new algorithm based on the Alternating Direction Method of Multipliers to adaptively balance these components.

Abstract

In this paper, we consider the conditional generation problem by guiding off-the-shelf unconditional diffusion models with differentiable loss functions in a plug-and-play fashion. While previous research has primarily focused on balancing the unconditional diffusion model and the guided loss through a tuned weight hyperparameter, we propose a novel framework that distinctly decouples these two components. Specifically, we introduce two variables ${x}$ and ${z}$, to represent the generated samples governed by the unconditional generation model and the guidance function, respectively. This decoupling reformulates conditional generation into two manageable subproblems, unified by the constraint ${x} = {z}$. Leveraging this setup, we develop a new algorithm based on the Alternating Direction Method of Multipliers (ADMM) to adaptively balance these components. Additionally, we establish the equivalence between the diffusion reverse step and the proximal operator of ADMM and provide a detailed convergence analysis of our algorithm under certain mild assumptions. Our experiments demonstrate that our proposed method ADMMDiff consistently generates high-quality samples while ensuring strong adherence to the conditioning criteria. It outperforms existing methods across a range of conditional generation tasks, including image generation with various guidance and controllable motion synthesis.

Figures (5)

• Figure 1: Illustrated results of ADMMDiff on diverse conditional generation tasks.ADMMDiff effectively guides the generation process of diffusion models using training-free guidance functions, producing high-quality samples that adhere closely to the specified conditions.
• Figure 2: Geometrical illustration of ADMM-based method. Compared with classic guided diffusion frameworks which directly perturbs reverse diffusion steps with guidance gradients, ADMM-based method decouples the guidance gradient from the reverse diffusion trajectory and allows more flexibility to explore guidance conditions.
• Figure 3: Qualitative comparison on CelebA-HQ in three conditional image synthesis tasks: (a) segmentation maps to human faces; (b) sketches to human faces; (c) text prompts to human faces. Our method offers comparable image quality and advantage in the degree of satisfaction of the conditions.
• Figure 4: Qualitative comparison on solving linear inverse problems using our method and DPS. Our method consistently achieve comparable image quality and advantage in the degree of satisfaction of the conditions.
• Figure 5: Qualitative comparison on controllable motion generation using trajectory guidance. Blue line indicates the path to follow. The results show that our method better follows the trajectory while being consistent with the motion prior and text prompt.

Theorems & Definitions (8)

• Proposition 1
• Theorem 1
• Theorem 2
• proof : Proof of Theorem \ref{['thm:1']}
• Proposition 2
• proof
• Lemma 1: robbins_1971_convergence
• proof : Proof of Theorem \ref{['thm: 2']}