A Noise is Worth Diffusion Guidance

TL;DR

<3-5 sentence high-level summary> NoiseRefine introduces a noise-space mapping approach to eliminate the need for guidance in diffusion-based image synthesis. By learning to map standard Gaussian noise to a guidance-free noise space via a lightweight network and training with Multistep Score Distillation, the method achieves high-quality unguided generation with about 2x–3x speedups compared to CFG-based baselines. The approach relies on diffusion inversion insights, emphasizes low-frequency components for layout formation, and uses a curated 50K-image-equivalent dataset with careful filtering and prompt variety. Empirical results across FID/IS, qualitative assessments, and a user study demonstrate competitive image quality and prompt adherence while reducing inference cost and memory consumption.

Abstract

Diffusion models excel in generating high-quality images. However, current diffusion models struggle to produce reliable images without guidance methods, such as classifier-free guidance (CFG). Are guidance methods truly necessary? Observing that noise obtained via diffusion inversion can reconstruct high-quality images without guidance, we focus on the initial noise of the denoising pipeline. By mapping Gaussian noise to `guidance-free noise', we uncover that small low-magnitude low-frequency components significantly enhance the denoising process, removing the need for guidance and thus improving both inference throughput and memory. Expanding on this, we propose \ours, a novel method that replaces guidance methods with a single refinement of the initial noise. This refined noise enables high-quality image generation without guidance, within the same diffusion pipeline. Our noise-refining model leverages efficient noise-space learning, achieving rapid convergence and strong performance with just 50K text-image pairs. We validate its effectiveness across diverse metrics and analyze how refined noise can eliminate the need for guidance. See our project page: https://cvlab-kaist.github.io/NoiseRefine/.

Figures (29)

• Figure 1: Effectiveness of NoiseRefine. Diffusion models often fail to generate high-quality images without guidance, such as classifier-free guidance (CFG) ho2022classifier. We propose NoiseRefine, a novel approach to improve image quality without use of guidance by learning to map initial random noise to a guidance-free noise space. Results are demonstrated using Stable Diffusion 2.1rombach2022high.
• Figure 2: Insight of NoiseRefine. We combine inversion methods song2020denoisingmeiri2023fixedgaribi2024renoise and guidance methods ho2022classifierahn2024selfhong2023improvingsadat2024nohong2024smoothedkarras2024guiding to establish a mapping between standard noise $x_T$ and guidance-free noise $x_{T}^{\text{Guide}}$.
• Figure 3: Analysis on the relationship between the initial Gaussian noise$x_T$and the guidance-free noise$x_{T}^{\text{Guide}}$. (a) shows the histogram of the absolute difference between $x_T$ and $x_{T}^{\text{Guide}}$. Here, 'Random' denotes the setting where the both noises are replaced with independent gaussian white noise. (b) presents the magnitude difference between the 2D Fourier-transformed frequency components of $\mathcal{F}(x_T)$ and $\mathcal{F}(x_{T}^{\text{Guide}})$. The difference between $x_T$ and $x_{T}^{\text{Guide}}$ is significantly smaller than in the random case, which mainly corresponds to the low-frequency components.
• Figure 4: Training pipeline. We propose a training methodology to learn a mapping from initial noise to guidance-free noise. Given an initial Gaussian noise $x_T$, the original diffusion model parameterized by $\theta$ generates an image $x_{0}^{\text{Guide}}$ using guidance ho2022classifierahn2024self. Noise refining model refines the initial noise $x_T$ to produce $\hat{x}_T = g_\phi(x_T)$, which is then input to the original model to generate an image $\hat{x}_0$ without guidance. By minimizing the distance between two images $d(x_{0}^{\text{Guide}}, \hat{x}_0)$, noise refining model effectively learns the desired mapping. Note that both noise refining model and original model also receive a prompt $c$ as input, though this is omitted here for simplicity.
• Figure 5: Comparison between noise optimization methods. We compare two methods to optimize a noise for target image generation. (a) illustrates direct optimization using inversion noise from the target image, while (b) shows optimization by minimizing the loss between denoised image and the target image. The rightmost column visualizes each optimized noise in a low-frequency area, indicating the similarity between the two noises.

Theorems & Definitions (2)

• Proposition 1
• Proposition 2