Frequency-Guided Posterior Sampling for Diffusion-Based Image Restoration

TL;DR

This paper addresses the gap in diffusion-based image restoration where approximations to the conditional likelihood can lead to large errors. It introduces Frequency Guided Posterior Sampling (FGPS), a method that applies a time varying low pass filter in the measurement domain and uses a data driven frequency curriculum to progressively incorporate higher frequencies during the reverse diffusion. A theoretical analysis under a Gaussian spectral model characterizes the approximation gap and shows FGPS can better approximate the true conditional score than prior methods, especially for high frequency content and challenging forward operators. Empirically, FGPS yields substantial improvements on motion deblurring, high pass deconvolution, and image dehazing on FFHQ and ImageNet, with efficient FFT based implementation and robust performance across datasets.

Abstract

Image restoration aims to recover high-quality images from degraded observations. When the degradation process is known, the recovery problem can be formulated as an inverse problem, and in a Bayesian context, the goal is to sample a clean reconstruction given the degraded observation. Recently, modern pretrained diffusion models have been used for image restoration by modifying their sampling procedure to account for the degradation process. However, these methods often rely on certain approximations that can lead to significant errors and compromised sample quality. In this paper, we provide the first rigorous analysis of this approximation error for linear inverse problems under distributional assumptions on the space of natural images, demonstrating cases where previous works can fail dramatically. Motivated by our theoretical insights, we propose a simple modification to existing diffusion-based restoration methods. Our approach introduces a time-varying low-pass filter in the frequency domain of the measurements, progressively incorporating higher frequencies during the restoration process. We develop an adaptive curriculum for this frequency schedule based on the underlying data distribution. Our method significantly improves performance on challenging image restoration tasks including motion deblurring and image dehazing.

Figures (17)

• Figure 1: We show the exact approximation gap for FGPS and DPS across diffusion timesteps when the data follows a power law in the frequency domain and the forward operator is a a high-pass filter (Dirac kernel minus a Gaussian kernel) of varying width. The width denotes the $\sigma$ of the Gaussian kernel. Note the $y$-axis is in log-scale.
• Figure 2: Even for linear inverse problems, SOTA diffusion-based methods (DPS, MCG, DSG) can struggle to solve inverse problems when the forward operator is convolution with a high-pass filter. In contrast, our method is still able to achieve a high quality reconstruction.
• Figure 3: Qualitative results on FFHQ and ImageNet datasets. The dotted red boxes highlight areas of the image where our method results in higher-quality reconstructions than DPS and DSG. Zoomed in versions can be found in the Appendix.
• Figure 4: Visualization of Transformed Measurements $y_t$ at different timesteps $t$, demonstrating our coarse-to-fine strategy.
• Figure 5: Qualitative Ablation Study: fixed low-pass filter vs. time-dependent low pass filter.

Theorems & Definitions (11)

• Lemma 1
• Theorem 2
• Corollary 3
• Lemma 4
• proof
• Lemma 5
• proof
• Lemma 6
• proof
• Theorem 7