Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Seeing Motion at Nighttime with an Event Camera

Haoyue Liu, Shihan Peng, Lin Zhu, Yi Chang, Hanyu Zhou, Luxin Yan

TL;DR

This work tackles nighttime dynamic-scene imaging with event cameras as an alternative to long-exposure frame-based methods. It identifies temporal trailing and spatial non-uniformity in nighttime events and proposes NER-Net, which couples Learnable Event Timestamps Calibration (LETC) with a Non-uniform Illumination Aware Module (NIAM) and an ETS-based trail suppression strategy. A new real paired dataset, RLED, provides pixel-aligned GTs for low-light events across 0.5–1000 lux, enabling robust learning. Experiments across RLED, DSEC-night, MVSEC-night show state-of-the-art performance and strong generalization, suggesting practical impact for nighttime autonomous driving and surveillance.

Abstract

We focus on a very challenging task: imaging at nighttime dynamic scenes. Most previous methods rely on the low-light enhancement of a conventional RGB camera. However, they would inevitably face a dilemma between the long exposure time of nighttime and the motion blur of dynamic scenes. Event cameras react to dynamic changes with higher temporal resolution (microsecond) and higher dynamic range (120dB), offering an alternative solution. In this work, we present a novel nighttime dynamic imaging method with an event camera. Specifically, we discover that the event at nighttime exhibits temporal trailing characteristics and spatial non-stationary distribution. Consequently, we propose a nighttime event reconstruction network (NER-Net) which mainly includes a learnable event timestamps calibration module (LETC) to align the temporal trailing events and a non-uniform illumination aware module (NIAM) to stabilize the spatiotemporal distribution of events. Moreover, we construct a paired real low-light event dataset (RLED) through a co-axial imaging system, including 64,200 spatially and temporally aligned image GTs and low-light events. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art methods in terms of visual quality and generalization ability on real-world nighttime datasets. The project are available at: https://github.com/Liu-haoyue/NER-Net.

Seeing Motion at Nighttime with an Event Camera

TL;DR

This work tackles nighttime dynamic-scene imaging with event cameras as an alternative to long-exposure frame-based methods. It identifies temporal trailing and spatial non-uniformity in nighttime events and proposes NER-Net, which couples Learnable Event Timestamps Calibration (LETC) with a Non-uniform Illumination Aware Module (NIAM) and an ETS-based trail suppression strategy. A new real paired dataset, RLED, provides pixel-aligned GTs for low-light events across 0.5–1000 lux, enabling robust learning. Experiments across RLED, DSEC-night, MVSEC-night show state-of-the-art performance and strong generalization, suggesting practical impact for nighttime autonomous driving and surveillance.

Abstract

We focus on a very challenging task: imaging at nighttime dynamic scenes. Most previous methods rely on the low-light enhancement of a conventional RGB camera. However, they would inevitably face a dilemma between the long exposure time of nighttime and the motion blur of dynamic scenes. Event cameras react to dynamic changes with higher temporal resolution (microsecond) and higher dynamic range (120dB), offering an alternative solution. In this work, we present a novel nighttime dynamic imaging method with an event camera. Specifically, we discover that the event at nighttime exhibits temporal trailing characteristics and spatial non-stationary distribution. Consequently, we propose a nighttime event reconstruction network (NER-Net) which mainly includes a learnable event timestamps calibration module (LETC) to align the temporal trailing events and a non-uniform illumination aware module (NIAM) to stabilize the spatiotemporal distribution of events. Moreover, we construct a paired real low-light event dataset (RLED) through a co-axial imaging system, including 64,200 spatially and temporally aligned image GTs and low-light events. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art methods in terms of visual quality and generalization ability on real-world nighttime datasets. The project are available at: https://github.com/Liu-haoyue/NER-Net.
Paper Structure (15 sections, 8 equations, 9 figures, 3 tables)

This paper contains 15 sections, 8 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Nighttime Event Signal Characteristics
  4. Real Low-light Event Dataset
  5. Trailing Events in Temporal Domain
  6. Non-uniform Illumination in Spatial Domain
  7. Nighttime Event Reconstruction Network
  8. Trail Prior Guided Event Representation
  9. Non-uniform Illumination Aware Module
  10. Training Details
  11. Experiments
  12. Datasets and Experimental Settings
  13. Quantitative and Qualitative Evaluation
  14. Ablation Study and Discussion
  15. Conclusion

Figures (9)

  • Figure 1: Features of the proposed RLED. (a) The implementation of our coaxial imaging system. (b) Distribution of illumination and scene of the proposed dataset. (c) Visualization of RLED, which collects low-light events aligned with high-quality images at the pixel level.
  • Figure 2: This figure illustrates the mechanism of trailing events. (a) Photoreceptor cut-off frequency decreases with decreasing illumination. (b) The response time of the photoreceptor increases as the cut-off frequency decreases, requiring more time($\Delta t_l$ > $\Delta t_h$) to reach the same voltage change $V_{log}$ at low illumination. (c) The increase in response time leads to the effect of trailing events.
  • Figure 3: The spatial characteristics of events at night. (a) The illuminance field under artificial nighttime lighting is non-uniform and exhibits significant variations with different distances from the light source. (b) Events are more densely distributed in the vicinity of the artificial light source. (c) Greater luminance difference between nearer the light source and the dark background, resulting in more events being triggered, and vice versa.
  • Figure 4: The overall architecture of the proposed Nighttime Event Reconstruction(NER) network, which can model the non-stationary spatiotemporal distribution of nighttime events. (a) NER contains a Learnable Event Timestamps Calibration (LETC) and a U-shaped image reconstruction network with Non-uniform Illumination Aware Module (NIAM) encoders. (b) LETC generates voxels with sharp edges by redistributing the weights of event timestamps within different voxel units. (c) NIAM performs local and global illumination sensing and regulation via the Local Adaptation Gate(LAG) and Global Context Block(GCB), respectively. By using a Spatiotemporal Aggregation Unit(SAU), NIAM can adaptively exploit and fuse multi-scale spatial information and long-term temporal information.
  • Figure 5: Method and visualization of Event Trail Suppression (ETS). (a) ETS is designed based on the event response characteristics in Section 3.2 with three conditions. (b) ETS aligns event timestamps to the correct position. (c) Events processed by ETS result in sharper edges.
  • ...and 4 more figures