Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DualDn: Dual-domain Denoising via Differentiable ISP

Ruikang Li, Yujin Wang, Shiqi Chen, Fan Zhang, Jinwei Gu, Tianfan Xue

TL;DR

This work proposes DualDn, a novel learning-based dual-domain denoising that achieves greater generalizability compared to most learning-based denoising methods, as it can adapt to different unseen noises, ISP parameters, and even novel ISP pipelines.

Abstract

Image denoising is a critical component in a camera's Image Signal Processing (ISP) pipeline. There are two typical ways to inject a denoiser into the ISP pipeline: applying a denoiser directly to captured raw frames (raw domain) or to the ISP's output sRGB images (sRGB domain). However, both approaches have their limitations. Residual noise from raw-domain denoising can be amplified by the subsequent ISP processing, and the sRGB domain struggles to handle spatially varying noise since it only sees noise distorted by the ISP. Consequently, most raw or sRGB domain denoising works only for specific noise distributions and ISP configurations. To address these challenges, we propose DualDn, a novel learning-based dual-domain denoising. Unlike previous single-domain denoising, DualDn consists of two denoising networks: one in the raw domain and one in the sRGB domain. The raw domain denoising adapts to sensor-specific noise as well as spatially varying noise levels, while the sRGB domain denoising adapts to ISP variations and removes residual noise amplified by the ISP. Both denoising networks are connected with a differentiable ISP, which is trained end-to-end and discarded during the inference stage. With this design, DualDn achieves greater generalizability compared to most learning-based denoising methods, as it can adapt to different unseen noises, ISP parameters, and even novel ISP pipelines. Experiments show that DualDn achieves state-of-the-art performance and can adapt to different denoising architectures. Moreover, DualDn can be used as a plug-and-play denoising module with real cameras without retraining, and still demonstrate better performance than commercial on-camera denoising. The project website is available at: https://openimaginglab.github.io/DualDn/

DualDn: Dual-domain Denoising via Differentiable ISP

TL;DR

This work proposes DualDn, a novel learning-based dual-domain denoising that achieves greater generalizability compared to most learning-based denoising methods, as it can adapt to different unseen noises, ISP parameters, and even novel ISP pipelines.

Abstract

Image denoising is a critical component in a camera's Image Signal Processing (ISP) pipeline. There are two typical ways to inject a denoiser into the ISP pipeline: applying a denoiser directly to captured raw frames (raw domain) or to the ISP's output sRGB images (sRGB domain). However, both approaches have their limitations. Residual noise from raw-domain denoising can be amplified by the subsequent ISP processing, and the sRGB domain struggles to handle spatially varying noise since it only sees noise distorted by the ISP. Consequently, most raw or sRGB domain denoising works only for specific noise distributions and ISP configurations. To address these challenges, we propose DualDn, a novel learning-based dual-domain denoising. Unlike previous single-domain denoising, DualDn consists of two denoising networks: one in the raw domain and one in the sRGB domain. The raw domain denoising adapts to sensor-specific noise as well as spatially varying noise levels, while the sRGB domain denoising adapts to ISP variations and removes residual noise amplified by the ISP. Both denoising networks are connected with a differentiable ISP, which is trained end-to-end and discarded during the inference stage. With this design, DualDn achieves greater generalizability compared to most learning-based denoising methods, as it can adapt to different unseen noises, ISP parameters, and even novel ISP pipelines. Experiments show that DualDn achieves state-of-the-art performance and can adapt to different denoising architectures. Moreover, DualDn can be used as a plug-and-play denoising module with real cameras without retraining, and still demonstrate better performance than commercial on-camera denoising. The project website is available at: https://openimaginglab.github.io/DualDn/
Paper Structure (13 sections, 5 equations, 11 figures, 4 tables)

This paper contains 13 sections, 5 equations, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Denoising in raw domain.
  4. Denoising in sRGB domain.
  5. Method
  6. Image Generalization
  7. Differentiable ISP
  8. Denoising in Dual-domain
  9. Experiments
  10. Evaluation on Synthetic Noise
  11. Evaluation on Real Noise
  12. Ablation Study
  13. Conclusion

Figures (11)

  • Figure 1: Illustrating the generalizability of our dual-domain denoising. The proposed denoising outperforms commercial denoising algorithms on two smartphone cameras under various ISOs. Notice that our denoising is only trained on synthetic images, without using any images from these cameras or their ISP pipelines during training.
  • Figure 2: Compare single-domain and dual-domain denoising. Noise in raw domain is sensor-specific and ISO-dependent, and ISP is device-related and user-preferred. Only denoising in dual-domain can properly deal with noise and ISP variations respectively.
  • Figure 3: The overall pipeline of DualDn. It consists of 3 key components: (a) image generalization with various noise, (b) dual-domain denoising with noise map fusion, and (c) differentiable ISP with corresponding EXIF data and variable ISP parameters.
  • Figure 4: Generate different noise and different ISP amplification ratios during training.
  • Figure 5: Testing denoising performance with 3 backbones at various noise levels $K$.
  • ...and 6 more figures