Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Deep Demosaicing with Spatially Downsampled Isotropic Networks

Cory Fan, Wenchao Zhang

TL;DR

This work tackles the challenge of efficient demosaicing on mobile devices by challenging the conventional wisdom of keeping isotropic networks at full resolution. It introduces JD3Net, a simple fully convolutional isotropic network that uses deliberate spatial downsampling, guided by a resolution-invariant entropy objective adapted from DeepMAD, to achieve superior efficiency without sacrificing accuracy. Through extensive experiments on real and synthetic demosaicing tasks (including JDD, Bayer, and HybridEVS patterns), downsampling yields both higher PSNR and significantly lower FLOPs compared with non-downsampled baselines. The results demonstrate that downsampling can broadly improve isotropic network performance in demosaicing, offering a practical path toward real-time mobile ISP pipelines with lightweight models.

Abstract

In digital imaging, image demosaicing is a crucial first step which recovers the RGB information from a color filter array (CFA). Oftentimes, deep learning is utilized to perform image demosaicing. Given that most modern digital imaging applications occur on mobile platforms, applying deep learning to demosaicing requires lightweight and efficient networks. Isotropic networks, also known as residual-in-residual networks, have been often employed for image demosaicing and joint-demosaicing-and-denoising (JDD). Most demosaicing isotropic networks avoid spatial downsampling entirely, and thus are often prohibitively expensive computationally for mobile applications. Contrary to previous isotropic network designs, this paper claims that spatial downsampling to a signficant degree can improve the efficiency and performance of isotropic networks. To validate this claim, we design simple fully convolutional networks with and without downsampling using a mathematical architecture design technique adapted from DeepMAD, and find that downsampling improves empirical performance. Additionally, empirical testing of the downsampled variant, JD3Net, of our fully convolutional networks reveals strong empirical performance on a variety of image demosaicing and JDD tasks.

Efficient Deep Demosaicing with Spatially Downsampled Isotropic Networks

TL;DR

This work tackles the challenge of efficient demosaicing on mobile devices by challenging the conventional wisdom of keeping isotropic networks at full resolution. It introduces JD3Net, a simple fully convolutional isotropic network that uses deliberate spatial downsampling, guided by a resolution-invariant entropy objective adapted from DeepMAD, to achieve superior efficiency without sacrificing accuracy. Through extensive experiments on real and synthetic demosaicing tasks (including JDD, Bayer, and HybridEVS patterns), downsampling yields both higher PSNR and significantly lower FLOPs compared with non-downsampled baselines. The results demonstrate that downsampling can broadly improve isotropic network performance in demosaicing, offering a practical path toward real-time mobile ISP pipelines with lightweight models.

Abstract

In digital imaging, image demosaicing is a crucial first step which recovers the RGB information from a color filter array (CFA). Oftentimes, deep learning is utilized to perform image demosaicing. Given that most modern digital imaging applications occur on mobile platforms, applying deep learning to demosaicing requires lightweight and efficient networks. Isotropic networks, also known as residual-in-residual networks, have been often employed for image demosaicing and joint-demosaicing-and-denoising (JDD). Most demosaicing isotropic networks avoid spatial downsampling entirely, and thus are often prohibitively expensive computationally for mobile applications. Contrary to previous isotropic network designs, this paper claims that spatial downsampling to a signficant degree can improve the efficiency and performance of isotropic networks. To validate this claim, we design simple fully convolutional networks with and without downsampling using a mathematical architecture design technique adapted from DeepMAD, and find that downsampling improves empirical performance. Additionally, empirical testing of the downsampled variant, JD3Net, of our fully convolutional networks reveals strong empirical performance on a variety of image demosaicing and JDD tasks.
Paper Structure (15 sections, 3 equations, 5 figures, 7 tables)

This paper contains 15 sections, 3 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tasks in Demosaicing
  4. Isotropic Networks for Demosaicing
  5. Zero-Shot Neural Architecture Search
  6. Methods
  7. Network Design
  8. Mathematical Architecture Design
  9. Experiments and Analysis
  10. Searching for JD3Net
  11. Real Image Unified Joint-Demosaicing-and-Denoising
  12. Synthetic Quad-Bayer HybridEVS Demosaicing
  13. Synthetic Bayer Demosaicing
  14. Ablation
  15. Conclusion

Figures (5)

  • Figure 1: PSNR vs. computational cost on unified joint-demosaicing-and-denoising on HDD (top) and Quad-Bayer HybridEVS demosaicing (bottom).
  • Figure 2: CFAs investigated in this paper. Networks often have to deal with challenging CFAs with missing information (HybridEVS CFA, for instance) or multiple CFA simultaneously.
  • Figure 3: Simplified-NAFBlock and JD3Net Architecture. (A) NAFBlock. SCA stands for Simple Channel Attention. Dconv stands for depthwise convolution. (B) Simplified-NAFBlock, which is the same as NAFBlock except for removal of SCA. (C) Our fully-convolutional JD3Net architecture. JD3Net is fully-convolutional and highly simple.
  • Figure 4: Qualitative Results for ESUM and JD3Net on HDD. All are ISO 3200 Nona Bayer. Demosaicing occurred on RAW images, but post-processing was done with a normal ISP pipeline to make the images visually normal. JD3Net produces more detailed and accurate images with less computational cost.
  • Figure 5: Qualitative Results for DemosaicFormer and JD3Net on Quad-Bayer HybridEVS demosaicing.