DemosaicFormer: Coarse-to-Fine Demosaicing Network for HybridEVS Camera

TL;DR

This work tackles demosaicing for HybridEVS cameras by introducing DemosaicFormer, a coarse-to-fine two-stage network that separates coarse demosaicing from defect-pixel correction. A novel Multi-Scale Gating Module enables efficient cross-scale feature fusion within a Transformer-based correction stage, while progressive training and data augmentation boost robustness. Joint training of both stages, guided by an $L_1$ loss, yields superior PSNR and SSIM performance, achieving top results in the MIPI 2024 Demosaic for HybridEVS Camera track. The proposed approach demonstrates strong potential for practical ISP pipelines in HybridEVS-enabled devices, delivering high-quality RGB reconstructions under challenging conditions and sensor defects.

Abstract

Hybrid Event-Based Vision Sensor (HybridEVS) is a novel sensor integrating traditional frame-based and event-based sensors, offering substantial benefits for applications requiring low-light, high dynamic range, and low-latency environments, such as smartphones and wearable devices. Despite its potential, the lack of Image signal processing (ISP) pipeline specifically designed for HybridEVS poses a significant challenge. To address this challenge, in this study, we propose a coarse-to-fine framework named DemosaicFormer which comprises coarse demosaicing and pixel correction. Coarse demosaicing network is designed to produce a preliminary high-quality estimate of the RGB image from the HybridEVS raw data while the pixel correction network enhances the performance of image restoration and mitigates the impact of defective pixels. Our key innovation is the design of a Multi-Scale Gating Module (MSGM) applying the integration of cross-scale features, which allows feature information to flow between different scales. Additionally, the adoption of progressive training and data augmentation strategies further improves model's robustness and effectiveness. Experimental results show superior performance against the existing methods both qualitatively and visually, and our DemosaicFormer achieves the best performance in terms of all the evaluation metrics in the MIPI 2024 challenge on Demosaic for Hybridevs Camera. The code is available at https://github.com/QUEAHREN/DemosaicFormer.

Figures (6)

• Figure 1: Illustration of two different patterns and demosacing task. (a) Quad Bayer pattern. (b) HybridEVS pattern. (c) Demosaic for HybridEVS Camera task refers to the conversion of HybridEVS pattern raw data into RGB images.
• Figure 2: The architecture of our proposed DemosaicFormer to demosaic the raw data captured by HybridEVS cameras.
• Figure 3: The structures of sub-modules in the main architecture.
• Figure 4: Train model using synthesized augmented data.
• Figure 5: Visual comparison results of Demosaic for Hybridevs Camera on the evaluation dataset of MIPI-challenge 2024 track. Note that brighter means bigger error.