Adaptive Denoising via GainTuning

TL;DR

GainTuning is proposed, in which CNN models pre-trained on large datasets are adaptively and selectively adjusted for individual test images, and is able to faithfully reconstruct the structure of catalytic nanoparticles from these data at extremely low signal-to-noise ratios.

Abstract

Deep convolutional neural networks (CNNs) for image denoising are usually trained on large datasets. These models achieve the current state of the art, but they have difficulties generalizing when applied to data that deviate from the training distribution. Recent work has shown that it is possible to train denoisers on a single noisy image. These models adapt to the features of the test image, but their performance is limited by the small amount of information used to train them. Here we propose "GainTuning", in which CNN models pre-trained on large datasets are adaptively and selectively adjusted for individual test images. To avoid overfitting, GainTuning optimizes a single multiplicative scaling parameter (the "Gain") of each channel in the convolutional layers of the CNN. We show that GainTuning improves state-of-the-art CNNs on standard image-denoising benchmarks, boosting their denoising performance on nearly every image in a held-out test set. These adaptive improvements are even more substantial for test images differing systematically from the training data, either in noise level or image type. We illustrate the potential of adaptive denoising in a scientific application, in which a CNN is trained on synthetic data, and tested on real transmission-electron-microscope images. In contrast to the existing methodology, GainTuning is able to faithfully reconstruct the structure of catalytic nanoparticles from these data at extremely low signal-to-noise ratios.

Figures (17)

• Figure 1: Proposed denoising paradigm. (a) Typically, CNNs are trained on a large dataset and evaluated directly on a test image. (b) Recent unsupervised methods perform training on a single test image. (c) We propose GainTuning, a framework which bridges the gap between both of these paradigms: a CNN pre-trained on a large training database is adapted to the test image.
• Figure 2: Denoising results for real-world data. (a) An experimentally-acquired atomic-resolution transmission electron microscope image of a CeO$_2$-supported Pt nanoparticle. The image has a very low signal to noise ratio (PSNR of $\approx 3 dB$). (b) Denoised image obtained using Self2Self self2self, which fails to reconstruct three atoms (blue arrow, second row). (c) Denoised image obtained via a CNN trained on a simulated dataset, where the pattern of the supporting atoms is not recovered faithfully (third row). (d) Denoised image obtained by adapting the CNN in (c) to the noisy test image in (a) using GainTuning. Both the nanoparticle and the support are recovered without artefacts. (e) Reference image, estimated by averaging $40$ different noisy images of the same nanoparticle.
• Figure 3: GainTuning achieves state-of-the-art performance. (Left) The average PSNR on two test set of generic natural images improves after GainTuning for different architectures across multiple noise levels. The CNNs are trained on generic natural images (BSD400). (Right) Histograms showing improvement in performance for each image in each of the two test sets at $\sigma=30$.
• Figure 4: GainTuning generalizes to out-of-distribution data. Average performance of a CNN trained to denoise at noise levels $\sigma \in [0, 55]$ improves significantly on test image with noise outside the training range, $\sigma=70, 80$ (top) and on images with different characteristics than training data (bottom) after GainTuning . Capability of GainTuning to generalize to out-of-distribution noise is comparable to that of Bias-Free CNN biasfree, which is an architecture explicitly designed to generalize to noise levels outside the training range, and to that of a denoiser trained with supervision at all noise levels. (Right) Histogram showing improvement in performance for each image in the test set. The improvement is substantial across most images, reaching nearly 12dB improvement in one example.
• Figure 5: Adaptation to new image content. (Top) A CNN pre-trained on piecewise constant images applied to a natural test image (a) oversmooths the image and blurs the details (b), but is able to recover more detail after applying GainTuning (c). (Bottom) The CNN estimates a denoised pixel (dot at the center of each image) as a linear combination of the noisy input pixels. The weighting functions (filters) of pre-trained CNN are more dispersed, consistent with the training set. However, after GainTuning, the weighting functions are more precisely targeted to the local features, resulting in a denoised image (c) with more details.