Frequency Enhancement for Image Demosaicking

TL;DR

The paper addresses demosaicking challenges in reconstructing high-frequency textures and mitigating color moiré by analyzing the frequency-domain behavior of CFA/demosaicked/ground-truth images and identifying aliasing near the Nyquist limit $f_N$. It introduces DFENet, a Dual-path Frequency Enhancement Network that uses two Fourier-domain frequency selectors to separately generate missing high-frequency information in the spatial path and suppress undesirable frequencies guided by CFA spectra in the frequency path, with multi-level frequency supervision and stagewise training. A new LineSet37 dataset with 37 hard-case images is proposed to stress-test demosaicking methods. Across benchmarks and LineSet37, DFENet achieves state-of-the-art performance, with extensive ablations confirming the effectiveness of frequency suppression, dual selectors, FFT-based processing, and stagewise optimization for robust high-frequency reconstruction.

Abstract

Recovering high-frequency textures in image demosaicking remains a challenging issue. While existing methods introduced elaborate spatial learning methods, they still exhibit limited performance. To address this issue, a frequency enhancement approach is proposed. Based on the frequency analysis of color filter array (CFA)/demosaicked/ground truth images, we propose Dual-path Frequency Enhancement Network (DFENet), which reconstructs RGB images in a divide-and-conquer manner through fourier-domain frequency selection. In DFENet, two frequency selectors are employed, each selecting a set of frequency components for processing along separate paths. One path focuses on generating missing information through detail refinement in spatial domain, while the other aims at suppressing undesirable frequencies with the guidance of CFA images in frequency domain. Multi-level frequency supervision with a stagewise training strategy is employed to further improve the reconstruction performance. With these designs, the proposed DFENet outperforms other state-of-the-art algorithms on different datasets and demonstrates significant advantages on hard cases. Moreover, to better assess algorithms' ability to reconstruct high-frequency textures, a new dataset, LineSet37, is contributed, which consists of 37 artificially designed and generated images. These images feature complex line patterns and are prone to severe visual artifacts like color moiré after demosaicking. Experiments on LineSet37 offer a more targeted evaluation of performance on challenging cases. The code and dataset are available at https://github.com/VelvetReverie/DFENet-demosaicking.

Figures (9)

• Figure 1: (a) CFA image under the Bayer pattern. (b) Schematic diagram of CFA image's spectrum. Replicas of the luminance spectrum (large triangle) do not overlap with each other but with chrominance components (small triangle).
• Figure 2: (a) Visualization of the demosaicked image produced from FlexISP, the ground truth image, the CFA image and their frequency spectrum. In comparison with the CFA image and the ground truth image, there are missing high frequencies (indicated by yellow arrow) and undesirable low frequencies (indicated by red arrow) in the spectrum of demosaicked image. (b) Left is the original signal, and right is the aliased signal. Red circles represent the sampling point. With a sampling rate lower than Nyquist frequency, the high-frequency signal is incorrectly reconstructed into a lower frequency, which explains the errors in the demosaicked image.
• Figure 3: The architecture of the proposed Dual-path Frequency Enhancement Network (DFENet). The network comprises $M$ cascaded Residual Frequency Enhancement Groups (RFEG). In RFEG, the features first go through $N_1$ Residual Channel Attention Blocks (RCAB) for coarse feature extraction and then into the Selective Frequency Enhancement (SFE) for refinement. A divide-and-conquer manner is employed through fourier-domain frequency selection to simultaneously generate missing details (the lower path in SFE) and suppress false frequencies (the upper path in SFE) under the guidance of the input CFA image. The refined features from the two paths are fused with a single convolution layer.
• Figure 4: Comparison of frequency spectrum between full-resolution and half-resolution image. Spectrums in (a) respectively correspond to the demosaicked image and the ground truth image, and those in (b) correspond to the downscaled demosaicked image and ground truth image with pixelunshuffle. The other three components obtained from pixelunshuffle are not shown as they are similar to the one presented.
• Figure 5: Images from the proposed dataset LineSet37.