LAN: Learning to Adapt Noise for Image Denoising

TL;DR

This work tackles image denoising under unseen noise by freezing a pretrained denoiser and learning a per-pixel input noise offset to align unseen noise with the distribution the model was trained on. The offset, trained with self-supervision, yields an adapted input $\mathbf{y}^u+\bm{\phi}^*$ that improves denoising performance across multiple backbones and real-world noise datasets, often outperforming full-network fine-tuning. By operating at the input level, LAN offers a data-efficient, per-image adaptation strategy that reduces overfitting risk and demonstrates favorable memory and compute efficiency compared with full model adaptation. The approach introduces a practical, orthogonal direction to enhance robustness to real-world noise without extensive retraining of denoisers, with potential for broader adoption in real-time restoration tasks.

Abstract

Removing noise from images, a.k.a image denoising, can be a very challenging task since the type and amount of noise can greatly vary for each image due to many factors including a camera model and capturing environments. While there have been striking improvements in image denoising with the emergence of advanced deep learning architectures and real-world datasets, recent denoising networks struggle to maintain performance on images with noise that has not been seen during training. One typical approach to address the challenge would be to adapt a denoising network to new noise distribution. Instead, in this work, we shift our focus to adapting the input noise itself, rather than adapting a network. Thus, we keep a pretrained network frozen, and adapt an input noise to capture the fine-grained deviations. As such, we propose a new denoising algorithm, dubbed Learning-to-Adapt-Noise (LAN), where a learnable noise offset is directly added to a given noisy image to bring a given input noise closer towards the noise distribution a denoising network is trained to handle. Consequently, the proposed framework exhibits performance improvement on images with unseen noise, displaying the potential of the proposed research direction. The code is available at https://github.com/chjinny/LAN

Figures (6)

• Figure 1: Overview of the motivation of our framework, Learning-to-Adapt-Noise (LAN). Instead of adapting a denoising network to unseen noise, LAN adapts the input noise itself by directly learning to offset the deviations between the unseen noise and the noise distribution a denoising network is trained on.
• Figure 2: Overview of conventional methods and our framework, Learning-to-Adapt-Noise (LAN, ours). (a) Pretraining of a denoising network is done with pairs of noisy-clean images, with standard L$_2$ loss. (b) Fine-tuning of a whole denoising network is done with only a given noisy image via self-supervision loss function, such as ZS-N2N mansour2023zero, to handle unseen noise in the image. (c) Learning-to-Adapt-Noise (LAN, ours) is similar to fine-tuning in that only a give noisy image is used with self-supervision loss. However, our method keeps the whole denoising network frozen and only adapts a given noisy image to handle unseen noise.
• Figure 3: Qualitative comparisons among different adaptation methods. Images are obtained with SIDD-pretrained Uformer. Full-trainable and LAN (Ours) finetuned the pretrained network via ZS-N2N for 20 iterations on Nam (first three rows) and PolyU (last two rows).
• Figure 4: Visualization of synthetic noisy images. Noisy image with train noise is a noisy image that is used for pretraining a denoising network (DnCNN). Noisy image with new noise contain a new noise that is different from pretraining. Adapted noisy image by LAN is a result of noise adaptation of noisy image with new noise. We observe that noise in the adapted noisy image becomes more similar to noisy image with train noise. Particularly, we observe that noise has been added to the top of the image, where there was previously no noise. As a result, LAN helps achieve better denoising performance.
• Figure 5: Plot of performance in PSNR over the number of adaptation iterations. Results are obtained with Uformer finetuned via ZS-N2N on Nam Dataset.