Towards Robust Event-based Networks for Nighttime via Unpaired Day-to-Night Event Translation

TL;DR

The paper addresses the challenge of domain shift from day to night in event-based vision by proposing an unpaired day-to-night event translation framework powered by a Diffusion GAN. It introduces a temporally disentangled encoder and a wavelet-based bottleneck, augmented with temporally shuffling contrastive regularization, and defines new evaluation metrics tailored to event data. The approach enables translating annotated day events into night-like events, improving downstream tasks such as nighttime semantic segmentation and object detection, and it demonstrates generalization on unseen datasets. The work offers practical code and shows potential for broader day-night robustness in event-based perception systems.

Abstract

Event cameras with high dynamic range ensure scene capture even in low-light conditions. However, night events exhibit patterns different from those captured during the day. This difference causes performance degradation when applying night events to a model trained solely on day events. This limitation persists due to a lack of annotated night events. To overcome the limitation, we aim to alleviate data imbalance by translating annotated day data into night events. However, generating events from different modalities challenges reproducing their unique properties. Accordingly, we propose an unpaired event-to-event day-to-night translation model that effectively learns to map from one domain to another using Diffusion GAN. The proposed translation model analyzes events in spatio-temporal dimension with wavelet decomposition and disentangled convolution layers. We also propose a new temporal contrastive learning with a novel shuffling and sampling strategy to regularize temporal continuity. To validate the efficacy of the proposed methodology, we redesign metrics for evaluating events translated in an unpaired setting, aligning them with the event modality for the first time. Our framework shows the successful day-to-night event translation while preserving the characteristics of events. In addition, through our translation method, we facilitate event-based modes to learn about night events by translating annotated day events into night events. Our approach effectively mitigates the performance degradation of applying real night events to downstream tasks. The code is available at https://github.com/jeongyh98/UDNET.

Figures (24)

• Figure 1: (a) The results of a segmentation model sun2022ess trained on day event data. Day events inference yields comparable results to the ground truth, while night events inference is unsatisfactory. (b) Due to a scarcity of nighttime data, the network is trained solely on day events and their labels. (c) We translate day events into night events to train the network and then (d) improve performance while applying real night events.
• Figure 2: Overall process of our proposed method. To build an unpaired day-to-night event translation model, we train a generator using various constraints. To preserve the spatio-temporal property of events, we propose a novel temporally shuffling contrastive regularization with spatial contrastive regularization.
• Figure 3: Details about our proposed generator. In the encoding process, we disentangle channels and merge them by their original time bin. Channels from each bin pass through the disentangled block, $\mathcal{G}_l$, and merge by the global block, $\mathcal{H}_l$, as shown in (a).
• Figure 4: Qualitative results of translated night events of each method. The red and blue dots mean positive and negative polarity events, respectively. The first column serves as an image aligned with input day events in the second column. Other columns show the results of each method. The last column visualizes reference night events.
• Figure 5: Visual comparisons of applying wavelet decomposition in the frequency branch. (a) shows the input day events. (b) and (c) show the results of applying the frequency branch passing HH and LL components through ResNet, respectively. (d) shows the result without the frequency branch. (e) shows an aligned image of the input.