Noise is All You Need: Solving Linear Inverse Problems by Noise Combination Sampling with Diffusion Models

TL;DR

This paper addresses solving linear inverse problems with pretrained diffusion models by introducing Noise Combination Sampling (NCS), which synthesizes an optimal noise vector as a linear combination of Gaussian noises to approximate the measurement score and condition the generation process without disrupting the diffusion path. The optimization has a closed-form solution $\bm{\gamma}^* = \dfrac{ \bm{c}^{\top} \mathbf{E}_t }{ \| \bm{c}^{\top} \mathbf{E}_t \|_2 }$, enabling robust performance with few diffusion steps and easy integration with existing solvers such as DPS and MPGD; DDCM is shown to be a special case of NCS. NCS unifies and improves diffusion-based inverse problem solvers, reduces hyperparameter tuning, and delivers improved stability and efficiency across tasks like inpainting, super-resolution, and deblurring, with a promising path to faster and more scalable diffusion-guided compression. The approach has practical impact by making diffusion-based conditioning more robust and computationally efficient, broadening their applicability to real-time or resource-constrained settings.

Abstract

Pretrained diffusion models have demonstrated strong capabilities in zero-shot inverse problem solving by incorporating observation information into the generation process of the diffusion models. However, this presents an inherent dilemma: excessive integration can disrupt the generative process, while insufficient integration fails to emphasize the constraints imposed by the inverse problem. To address this, we propose \emph{Noise Combination Sampling}, a novel method that synthesizes an optimal noise vector from a noise subspace to approximate the measurement score, replacing the noise term in the standard Denoising Diffusion Probabilistic Models process. This enables conditional information to be naturally embedded into the generation process without reliance on step-wise hyperparameter tuning. Our method can be applied to a wide range of inverse problem solvers, including image compression, and, particularly when the number of generation steps $T$ is small, achieves superior performance with negligible computational overhead, significantly improving robustness and stability.

Figures (11)

• Figure 1: An illustration showing the difference between exact approximation methods and NCS. The intervention, i.e., the measurement score in the existing methods pushes the trajectory off the manifold $\mathcal{M}_{t-1}$ of $\bm{x}_{t-1}$. In contrast, NCS embeds the intervention into the optimal noise within an ellipsoidal subspace, defined by the span of the noise codebook. This allows NCS to naturally preserve both the position of $\bm{x}_{t-1}$ on its manifold and the consistency of the diffusion process.
• Figure 2: Comparison of DPS and NCS-DPS across four inverse problems under varying sampling steps. NCS-DPS yields clearer details and greater stability, especially at small step counts.
• Figure 3: The comparison of the selected noise according to the original image. We choose a downsampled 64x64 image as the target image.
• Figure 4: Comparison of the compression efficiency of NCS-MPGD and DDCM. For $T=1000$, we choose $K=32768$, $m=12$, $C=8$. For $T=100$, we choose $K=32768$, $m=2$, $C=0$. Our proposal get equivalent performance in fewer time.
• Figure 5: Influence of iteration and codebook size. We conduct the experiment on the inpainting task (${\sigma} = 0$). The heatmap shows the improvement of the NCS-DPS compared to the DPS methods.

Theorems & Definitions (10)

• Theorem 1: Noise Combination Sampling
• Definition 1: NCS-DPS
• Definition 2: NCS-MPGD
• Definition 3: DDCM under the NCS Framework
• Theorem 2: Optimal Noise Combination
• Lemma 1: Gaussianity of unit-norm combinations
• proof
• proof
• Theorem 3: Equivalence of NCS Formulations
• proof