CARD: Correlation Aware Restoration with Diffusion

TL;DR

This work introduces CARD, a training-free method that handles spatially correlated noise in image restoration by whitening observations and applying noise-whitened diffusion updates, extending diffusion priors to realistic sensor noise. It frames CARD as a drop-in extension to DDRM, maintaining closed-form sampling efficiency while accommodating correlated noise through a covariance-based whitening transform. To evaluate real-world performance, the authors present CIN-D, a large rolling-shutter dataset with varied illumination and noise levels, enabling robust testing beyond i.i.d. assumptions. Empirically, CARD delivers consistent improvements over state-of-the-art denoising, deblurring, and super-resolution methods on both synthetic correlated-noise benchmarks and CIN-D, with demonstrated robustness and cross-sensor covariance generalization. This approach promises practical applicability across diverse imaging systems with spatially correlated noise, and CIN-D provides a valuable benchmark for future correlated-noise restoration research.

Abstract

Denoising diffusion models have achieved state-of-the-art performance in image restoration by modeling the process as sequential denoising steps. However, most approaches assume independent and identically distributed (i.i.d.) Gaussian noise, while real-world sensors often exhibit spatially correlated noise due to readout mechanisms, limiting their practical effectiveness. We introduce Correlation Aware Restoration with Diffusion (CARD), a training-free extension of DDRM that explicitly handles correlated Gaussian noise. CARD first whitens the noisy observation, which converts the noise into an i.i.d. form. Then, the diffusion restoration steps are replaced with noise-whitened updates, which inherits DDRM's closed-form sampling efficiency while now being able to handle correlated noise. To emphasize the importance of addressing correlated noise, we contribute CIN-D, a novel correlated noise dataset captured across diverse illumination conditions to evaluate restoration methods on real rolling-shutter sensor noise. This dataset fills a critical gap in the literature for experimental evaluation with real-world correlated noise. Experiments on standard benchmarks with synthetic correlated noise and on CIN-D demonstrate that CARD consistently outperforms existing methods across denoising, deblurring, and super-resolution tasks.

Figures (16)

• Figure 1: Play your CARDs right. Image sensors, particularly rolling-shutter ones, suffer from correlated noise. We propose CARD, a training-free diffusion-based method for solving inverse problems with correlated noise. CARD requires a covariance matrix estimate and a pre-trained diffusion model, and can solve linear inverse problems such as (top row) denoising, (middle row) deblurring, and (bottom row) super-resolution.
• Figure 1: CIN-D Dataset Examples. Images of the same scenes captured at different noise levels using a FLIR Blackfly S BFS-U3-63S4C-C color camera. Noise levels decrease from top to bottom through controlled variations in sensor gain and exposure time, while scene content and viewing geometry remain fixed. This illustrates the progressive degradation in image quality that our dataset captures for benchmarking restoration algorithms under realistic correlated noise conditions.
• Figure 2: Correlated noise in real cameras. Modern digital cameras, particularly the rolling shutter ones, have strongly correlated noise. Here we show dark images for a machine vision, and a mirrorless SLR camera, both equipped with rolling shutter CMOS sensors, and displaying strong spatial correlations in noise. This observation motivates CARD and its applications for solving inverse problems with correlated noise.
• Figure 2: Hardware setup. Camera and lens assembly used for CIN-D acquisition. A FLIR Blackfly S BFS-U3-63S4C-C camera is paired with a fixed-focal-length C-mount lens and rigidly mounted on an optical bench for indoor scene capture.
• Figure 3: CARD approach. CARD is a two-step training-free approach to solving image-based inverse problems corrupted by correlated noise, as often found in real cameras. In step 1, we whiten the measurement using a known covariance matrix. In step 2, we apply noise-whitened updates to a pre-trained unconditional diffusion model to solve the whitened inverse problem.