Towards Closing the Domain Gap with Event Cameras

TL;DR

The paper investigates the domain gap caused by day-night lighting in end-to-end autonomous driving and compares frame-based grayscale cameras with event cameras (DVS). Using the DDD20 dataset, it trains and evaluates models on day- and night-biased data, employing careful preprocessing, augmentation, and a ResNet-50 backbone to assess cross-domain robustness. Results show that DVS maintains more consistent performance across lighting conditions and incurs smaller domain-shift penalties than APS, offering superior cross-domain baselines, though not a complete closure of the gap. The work supports incorporating event cameras into sensor arrays to enhance reliability under varying illumination in autonomous robotics.

Abstract

Although traditional cameras are the primary sensor for end-to-end driving, their performance suffers greatly when the conditions of the data they were trained on does not match the deployment environment, a problem known as the domain gap. In this work, we consider the day-night lighting difference domain gap. Instead of traditional cameras we propose event cameras as a potential alternative which can maintain performance across lighting condition domain gaps without requiring additional adjustments. Our results show that event cameras maintain more consistent performance across lighting conditions, exhibiting domain-shift penalties that are generally comparable to or smaller than grayscale frames and provide superior baseline performance in cross-domain scenarios.

Figures (5)

• Figure 1: The data split methodology we use to create our day biased and night biased training sets. Our process creates training sets that intentionally suffer from a domain gap, while our test sets are purely from one lighting condition, to test the models' ability to generalize across the domain gap.
• Figure 2: Flowchart of the pruning process, excludes manual trimming of the beginning and end of the recordings.
• Figure 3: Our modified Resnet-50 architecture, with an ImageNet initialization. The input layer is modified to accept 1-channel images, and a Tanh activation function is added to the head to match the $[-1, 1]$ range of the scaled steering angle output.
• Figure 4: Comparison of DVS and APS sensor data under different lighting conditions. (a) and (c), as well as (b) and (d) were captured simultaneously. The day and night images were chosen to match similar sections of road, with the first row showing a road near a hillside, the second row passing groups of trees on the left, and the third row featuring open highway. Inspecting (a) and (b), it is seen that the event-framed DVS data has little difference across the lighting conditions. This parallel is not repeated between (c) and (d), which diverge significantly. This is consistent with the day/night data profiles of the sensors as shown in Table \ref{['tab:aps_dvs_day_night_stats']}.
• Figure 5: Comparison across sensors and lighting conditions. The y-axis represents values in degrees of steering. Subfigures (a) and (b) as well as (c) and (d) are from the same recording. The overall better DVS performance, as well as the superior cross-domain performance of the DVS over the APS is visible.