Noise2Noise: Learning Image Restoration without Clean Data

TL;DR

Noise2Noise shows that high-quality image restoration can be learned from corrupted data alone by leveraging the fact that corrupted targets preserve the underlying clean signal in expectation under common losses. The authors provide a theoretical justification and validate it across Gaussian, Poisson, Bernoulli, Monte Carlo rendering, and MRI undersampling tasks, demonstrating performance on par with clean-target training in many cases. Empirically, training with noisy targets often matches or surpasses clean-target training and offers substantial practical advantages, such as faster data generation (e.g., Monte Carlo rendering) and reduced data-collection burden. This work broadens the data-efficiency frontier for learned restoration and suggests new directions for combining forward-model information with corrupted-data training.

Abstract

We apply basic statistical reasoning to signal reconstruction by machine learning -- learning to map corrupted observations to clean signals -- with a simple and powerful conclusion: it is possible to learn to restore images by only looking at corrupted examples, at performance at and sometimes exceeding training using clean data, without explicit image priors or likelihood models of the corruption. In practice, we show that a single model learns photographic noise removal, denoising synthetic Monte Carlo images, and reconstruction of undersampled MRI scans -- all corrupted by different processes -- based on noisy data only.

Figures (9)

• Figure 1: Denoising performance ($dB$ in Kodak dataset) as a function of training epoch for additive Gaussian noise. (a) For i.i.d. (white) Gaussian noise, clean and noisy targets lead to very similar convergence speed and eventual quality. (b) For brown Gaussian noise, we observe that increased inter-pixel noise correlation (wider spatial blur; one graph per bandwidth) slows convergence down, but eventual performance remains close. (c) Effect of different allocations of a fixed capture budget to noisy vs. clean examples (see text).
• Figure 2: Example results for Gaussian, Poisson, and Bernoulli noise. Our result was computed by using noisy targets --- the corresponding result with clean targets is omitted because it is virtually identical in all three cases, as discussed in the text. A different comparison method is used for each noise type.
• Figure 3: Removing random text overlays corresponds to seeking the median pixel color, accomplished using the $L_1$ loss. The mean ($L_2$ loss) is not the correct answer: note shift towards mean text color. Only corrupted images shown during training.
• Figure 4: For random impulse noise, the approx. mode-seeking $L_0$ loss performs better than the mean ($L_2$) or median ($L_1$) seeking losses.
• Figure 5: PSNR of noisy-target training relative to clean targets with a varying percentage of target pixels corrupted by RGB impulse noise. In this test a separate network was trained for each corruption level, and the graph was averaged over the Kodak dataset.