Near-infrared Image Deblurring and Event Denoising with Synergistic Neuromorphic Imaging

TL;DR

This work tackles nighttime low-light imaging by jointly leveraging NIR and event camera data to suppress motion blur and event noise. It introduces the synergistic neuromorphic imaging framework and the MDEDNet architecture, featuring a Spectral Consistency Enhancement (SCE) module and a Cross-modal Multi-order Interaction (CMI) module to fuse cross-spectral information effectively. The authors formulate a unified task with equations for motion deblurring and event denoising, propose a dedicated CSDD dataset, and demonstrate superior performance over state-of-the-art methods on synthetic and real data, with ablations validating the contributions of SCE and CMI. The results demonstrate that cross-modal fusion driven by spectral consistency and multi-order gradients can substantially improve NIR deblurring and event denoising, enabling high-fidelity low-light imaging and neuromorphic reasoning in practical settings.

Abstract

The fields of imaging in the nighttime dynamic and other extremely dark conditions have seen impressive and transformative advancements in recent years, partly driven by the rise of novel sensing approaches, e.g., near-infrared (NIR) cameras with high sensitivity and event cameras with minimal blur. However, inappropriate exposure ratios of near-infrared cameras make them susceptible to distortion and blur. Event cameras are also highly sensitive to weak signals at night yet prone to interference, often generating substantial noise and significantly degrading observations and analysis. Herein, we develop a new framework for low-light imaging combined with NIR imaging and event-based techniques, named synergistic neuromorphic imaging, which can jointly achieve NIR image deblurring and event denoising. Harnessing cross-modal features of NIR images and visible events via spectral consistency and higher-order interaction, the NIR images and events are simultaneously fused, enhanced, and bootstrapped. Experiments on real and realistically simulated sequences demonstrate the effectiveness of our method and indicate better accuracy and robustness than other methods in practical scenarios. This study gives impetus to enhance both NIR images and events, which paves the way for high-fidelity low-light imaging and neuromorphic reasoning.

Figures (18)

• Figure 1: Comparison examples of shared view with NIR image and events. A single NIR and event camera is often accompanied by motion blur and noise in low-light environments. Our synergistic scheme can reconstruct the sharp NIR image and clean events.
• Figure 2: Architecture of the proposed MDEDNet, which consists of a dual-branch subnetwork, a spectral consistency enhancement (SCE) module, and a cross-modal multi-order interaction (CMI) module. MDEDNet takes a blurry NIR image and noisy events as input, and through collaborative enhancement, simultaneously outputs a sharp image and clean events.
• Figure 3: (a)-(d) represent events, NIR image, first-order and second-order gradient, respectively, along with their histograms.
• Figure 4: (a) The synergistic imaging system with a NIR camera and an event camera. (b) and (c) Two imaging samples with raw collected NIR images and corresponding noisy events. Notably, the dashed line area in sample #2 highlights spectral inconsistencies between the NIR and the events.
• Figure 5: Comparison results of our deblurring method with others on the synthetic dataset.