Guidance with Spherical Gaussian Constraint for Conditional Diffusion

TL;DR

Diffusion with Spherical Gaussian constraint (DSG) effectively constrains the guidance step within the intermediate data manifold through optimization and enables the use of larger guidance steps.

Abstract

Recent advances in diffusion models attempt to handle conditional generative tasks by utilizing a differentiable loss function for guidance without the need for additional training. While these methods achieved certain success, they often compromise on sample quality and require small guidance step sizes, leading to longer sampling processes. This paper reveals that the fundamental issue lies in the manifold deviation during the sampling process when loss guidance is employed. We theoretically show the existence of manifold deviation by establishing a certain lower bound for the estimation error of the loss guidance. To mitigate this problem, we propose Diffusion with Spherical Gaussian constraint (DSG), drawing inspiration from the concentration phenomenon in high-dimensional Gaussian distributions. DSG effectively constrains the guidance step within the intermediate data manifold through optimization and enables the use of larger guidance steps. Furthermore, we present a closed-form solution for DSG denoising with the Spherical Gaussian constraint. Notably, DSG can seamlessly integrate as a plugin module within existing training-free conditional diffusion methods. Implementing DSG merely involves a few lines of additional code with almost no extra computational overhead, yet it leads to significant performance improvements. Comprehensive experimental results in various conditional generation tasks validate the superiority and adaptability of DSG in terms of both sample quality and time efficiency.

Figures (20)

• Figure 1: DSG mitigates manifold deviation problem by introducing Spherical Gaussian constraint without relying on the linear manifold assumption. Simultaneously, DSG enables the use of larger step sizes, significantly reducing inference time while enhancing sample quality. The integration of DSG into existing training-free CDMs only incurs a few additional lines of code. Here we briefly show the performance of integrating DSG into recent CDMs, such as DPS dps for Inpainting, Super Resolution, Gaussian Deblurring, UGD Universal for Segmentation-text Guidance, and Freedom freedom for Style Guidance tasks. DSG imposes nearly negligible computational overhead while delivering substantial performance enhancements.
• Figure 2: Illustration of the manifold deviation problem in DPS (left) and a schematic overview of how our DSG restricts the guidance within the manifold (right). The red circular ring represents the concentration region of samples under deviated conditional guidance, the blue circular ring indicates the concentration region of samples under accurate conditional guidance, and the green circular ring represents the concentration region of samples without condition.
• Figure 3: Result in solving three linear inverse problems (Inpainting, Super-resolution, Gaussian deblurring) using DSG using the pre-trained FFHQ diffusion model.
• Figure 4: Qualitative results of Style Guidance using pre-trained Stable Diffusion.
• Figure 5: Qualitative result of Text-Segmentation Guidance using 500 denoising steps with Stable Diffusion.

Theorems & Definitions (5)

• Definition 4.1
• Proposition 4.2
• Proposition 4.3
• Proposition 1.1
• Proposition 1.2