Non-Uniform Exposure Imaging via Neuromorphic Shutter Control

TL;DR

The paper tackles motion-induced blur and noise in non-uniform exposure imaging by introducing a neuromorphic shutter control (NSC) system that uses an event camera to monitor real-time scene motion and adapt shutter timing, combined with a self-supervised event-based image denoising (SEID) framework to stabilize SNR without ground-truth data. NSC employs Global Event Accumulation (GEA) and Pyramid Event Accumulation (PEA) to produce motion measures, guiding exposure, and SEID uses two Event-based Image Prediction modules and a TripletFusion network with self-supervised losses to reconstruct high-quality frames. The authors validate on synthetic Vimeo-Triplet data and a real-world Neuromorphic Exposure Dataset (NED), achieving state-of-the-art PSNR/SSIM (e.g., NSC_p+SEID ≈ $39.26$ dB PSNR and $0.983$ SSIM) and improved feature tracking, while demonstrating real-time operation with latency under $1$ ms. This approach offers a practical path to robust imaging in challenging environments and dynamic scenes, enabling high-SNR, sharp frames under varying motion and illumination conditions.

Abstract

By leveraging the blur-noise trade-off, imaging with non-uniform exposures largely extends the image acquisition flexibility in harsh environments. However, the limitation of conventional cameras in perceiving intra-frame dynamic information prevents existing methods from being implemented in the real-world frame acquisition for real-time adaptive camera shutter control. To address this challenge, we propose a novel Neuromorphic Shutter Control (NSC) system to avoid motion blurs and alleviate instant noises, where the extremely low latency of events is leveraged to monitor the real-time motion and facilitate the scene-adaptive exposure. Furthermore, to stabilize the inconsistent Signal-to-Noise Ratio (SNR) caused by the non-uniform exposure times, we propose an event-based image denoising network within a self-supervised learning paradigm, i.e., SEID, exploring the statistics of image noises and inter-frame motion information of events to obtain artificial supervision signals for high-quality imaging in real-world scenes. To illustrate the effectiveness of the proposed NSC, we implement it in hardware by building a hybrid-camera imaging prototype system, with which we collect a real-world dataset containing well-synchronized frames and events in diverse scenarios with different target scenes and motion patterns. Experiments on the synthetic and real-world datasets demonstrate the superiority of our method over state-of-the-art approaches.

Figures (14)

• Figure 1: Comparisons of our neuromorphic exposure imaging system (left) with the conventional exposure setups (right). Our system contains two parts, i.e., Neuromorphic Shutter Control (NSC) utilizing the event camera to monitor the scene motion information and adjust the camera shutter in real time, and Self-supervised Event-based Image Denoising (SEID) stabilizing high SNR for each frame. Green solid lines indicate the process of image denoising and blue dashed lines indicate the generation of the self-supervised signals in SEID.
• Figure 2: (a) Pyramid Event Accumulation (PEA) is designed for the local motion. (b) Demonstration of the patch-wise counting for PEA with the parameters $s$ and $w$. $R$ is a threshold value defined in \ref{['eq:m']}.
• Figure 3: Comparison of the NSC results with two strategies sharing a same threshold $R$. (a) The instant frame with the local motion scene is disturbed by severe noise. (b) NSC with the GEA can significantly suppress noise, while the local blur may arise. (c) NSC with the PEA enhances the sensitivity to the local motion and achieves less severe blurs.
• Figure 4: (a) Overview of our Self-supervised Event-based Image Denoising (SEID) framework. (b) Details of the Event-based Image Prediction (EIP) module.
• Figure 5: Our prototype system and the data streams and external trigger streams among cameras and the development board.