Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How to Best Combine Demosaicing and Denoising?

Yu Guo, Qiyu Jin, Jean-Michel Morel, Gabriele Facciolo

TL;DR

This work tackles the joint problem of demosaicing and denoising in raw CFA images, asking whether to denoise before or after demosaicing. It introduces a flexible DN1&DM&DN2 framework and optimizes its four hyperparameters with CMA-ES to maximize CPSNR, establishing that for moderate noise the DM&1.5DN scheme (demosaicing followed by denoising with denoiser input scaled to 1.5σ) is near-optimal and far more practical than end-to-end heavy CNNs. A statistical analysis of demosaiced noise explains the success of the 1.5x noise policy by showing how luminance and chrominance components behave differently after demosaicing, with a decorrelated YC1C2 space offering insight into noise filtering strategies. Extensive experiments on simulated (Imax, Kodak) and real (SIDD, DND) datasets demonstrate that DM&1.5DN often yields higher CPSNR and better perceptual quality than DN&DM or DM&DN, while CMA-ES-optimized DN1&DM&DN2 provides the best performance at high computational cost. The results support using DM&1.5DN in practical, edge-friendly pipelines and provide theoretical guidance for designing light-weight, domain-robust raw denoising solutions, with deep learning remains influential but less portable for devices with limited resources.

Abstract

Image demosaicing and denoising play a critical role in the raw imaging pipeline. These processes have often been treated as independent, without considering their interactions. Indeed, most classic denoising methods handle noisy RGB images, not raw images. Conversely, most demosaicing methods address the demosaicing of noise free images. The real problem is to jointly denoise and demosaic noisy raw images. But the question of how to proceed is still not yet clarified. In this paper, we carry-out extensive experiments and a mathematical analysis to tackle this problem by low complexity algorithms. Indeed, both problems have been only addressed jointly by end-to-end heavy weight convolutional neural networks (CNNs), which are currently incompatible with low power portable imaging devices and remain by nature domain (or device) dependent. Our study leads us to conclude that, with moderate noise, demosaicing should be applied first, followed by denoising. This requires a simple adaptation of classic denoising algorithms to demosaiced noise, which we justify and specify. Although our main conclusion is ``demosaic first, then denoise'', we also discover that for high noise, there is a moderate PSNR gain by a more complex strategy: partial CFA denoising followed by demosaicing, and by a second denoising on the RGB image. These surprising results are obtained by a black-box optimization of the pipeline, which could be applied to any other pipeline. We validate our results on simulated and real noisy CFA images obtained from several benchmarks.

How to Best Combine Demosaicing and Denoising?

TL;DR

This work tackles the joint problem of demosaicing and denoising in raw CFA images, asking whether to denoise before or after demosaicing. It introduces a flexible DN1&DM&DN2 framework and optimizes its four hyperparameters with CMA-ES to maximize CPSNR, establishing that for moderate noise the DM&1.5DN scheme (demosaicing followed by denoising with denoiser input scaled to 1.5σ) is near-optimal and far more practical than end-to-end heavy CNNs. A statistical analysis of demosaiced noise explains the success of the 1.5x noise policy by showing how luminance and chrominance components behave differently after demosaicing, with a decorrelated YC1C2 space offering insight into noise filtering strategies. Extensive experiments on simulated (Imax, Kodak) and real (SIDD, DND) datasets demonstrate that DM&1.5DN often yields higher CPSNR and better perceptual quality than DN&DM or DM&DN, while CMA-ES-optimized DN1&DM&DN2 provides the best performance at high computational cost. The results support using DM&1.5DN in practical, edge-friendly pipelines and provide theoretical guidance for designing light-weight, domain-robust raw denoising solutions, with deep learning remains influential but less portable for devices with limited resources.

Abstract

Image demosaicing and denoising play a critical role in the raw imaging pipeline. These processes have often been treated as independent, without considering their interactions. Indeed, most classic denoising methods handle noisy RGB images, not raw images. Conversely, most demosaicing methods address the demosaicing of noise free images. The real problem is to jointly denoise and demosaic noisy raw images. But the question of how to proceed is still not yet clarified. In this paper, we carry-out extensive experiments and a mathematical analysis to tackle this problem by low complexity algorithms. Indeed, both problems have been only addressed jointly by end-to-end heavy weight convolutional neural networks (CNNs), which are currently incompatible with low power portable imaging devices and remain by nature domain (or device) dependent. Our study leads us to conclude that, with moderate noise, demosaicing should be applied first, followed by denoising. This requires a simple adaptation of classic denoising algorithms to demosaiced noise, which we justify and specify. Although our main conclusion is ``demosaic first, then denoise'', we also discover that for high noise, there is a moderate PSNR gain by a more complex strategy: partial CFA denoising followed by demosaicing, and by a second denoising on the RGB image. These surprising results are obtained by a black-box optimization of the pipeline, which could be applied to any other pipeline. We validate our results on simulated and real noisy CFA images obtained from several benchmarks.
Paper Structure (9 sections, 8 equations, 14 figures, 13 tables)

This paper contains 9 sections, 8 equations, 14 figures, 13 tables.

Table of Contents

  1. The demosaicing and denoising pipeline
  2. Pipeline optimization and analysis
  3. Analysis of
  4. Experimental evaluation
  5. Evaluation of and strategies on simulated images
  6. Evaluation of and strategies on real image datasets
  7. The strategy for raw image denoising
  8. Time consumption and generalizability
  9. Conclusion

Figures (14)

  • Figure 1: Image details at $\sigma = 20$. The lower row is the reconstructed image, and the upper row is the difference between the reconstructed image and ground truth. $DN$: cfaBM3D or CBM3D denoising; $DM$: RCNN demosaicing. $1.5DN$ means that if the noise level is $\sigma$, the input noise level parameter of denoising method $DN$ is $\sigma_2=1.5{{\sigma}}$.
  • Figure 2: The framework used for denoising before demosaicing using an RGB denoiser. The Bayer CFA image is split in two half resolution RGB images, each one with a different green. Both RGB images are denoised independently. Then the pixels of both results are recombined into a denoised Bayer CFA image. The last step consists in applying a demosaicing algorithm.
  • Figure 3: Generic raw image processing pipeline. This pipeline structure allows for an arbitrary order between $DN$ and $DM$ and sets free their parameters. We use the CMA-ES algorithm to optimize the parameter $\alpha$, $\beta$, $\sigma_1$, $\sigma_2$ in the pipeline.
  • Figure 4: A visual representation of the process in Figure \ref{['cma_pipeline']}, where the noise level is $\sigma = 60$. The parameter are $\alpha= 0.90$, $\beta= 0.99$, $\sigma_1= 34.50$, $\sigma_2= 54.42$. Since $\beta$ is always close to 1 in the pipeline, the visual difference between Color Denoising and the $\beta$ linear combination is not significant.
  • Figure 5: Evolution of the result of iterating CMA-ES when optimizing the parameters $\alpha, \beta, \sigma_1, \sigma_2$ of the processing pipeline.
  • ...and 9 more figures

Theorems & Definitions (1)

  • Definition 3.1