Sensor Generalization for Adaptive Sensing in Event-based Object Detection via Joint Distribution Training

TL;DR

This paper provides readers with an in-depth understanding of how intrinsic parameters affect the performance of a model trained on event data, specifically for object detection and uses the findings to expand the capabilities of the downstream model towards sensor-agnostic robustness.

Abstract

Bio-inspired event cameras have recently attracted significant research due to their asynchronous and low-latency capabilities. These features provide a high dynamic range and significantly reduce motion blur. However, because of the novelty in the nature of their output signals, there is a gap in the variability of available data and a lack of extensive analysis of the parameters characterizing their signals. This paper addresses these issues by providing readers with an in-depth understanding of how intrinsic parameters affect the performance of a model trained on event data, specifically for object detection. We also use our findings to expand the capabilities of the downstream model towards sensor-agnostic robustness.

Figures (4)

• Figure 1: Overview of a sensor control loop. We aim to dynamically adapt the sensor characteristics based on energy utilization and task performance feedback. In this work, we investigate and mitigate the undesired effects of dynamically varying spike transduction on task performance, i.e., object detection.
• Figure 2: Examples of two scenes with varying configurations which produce high and low event densities, respectively. The black and white points represent negative and positive events, respectively, accumulated over a 50-ms time period. The gray spaces indicate the occurrence of no events within this timeframe at the corresponding pixel locations. The corresponding instance segmentation map is provided for reference to the actual vehicular locations.
• Figure 3: Simplified overview of the design and positioning of the distinct test sets in the parameter space of two parameters - the field of view and negative threshold. The training set and the first test set consist of distinct configurations varying along each dimension. The second test set consists of configurations obtained by interpolating along each dimension, whereas the third set is obtained by combining seen parameters into unique configurations. The fourth set represents the unseen parameter space for each parameter.
• Figure 4: Comparison of results between the different models - RVT-B and SSMS-B trained on $\mathcal{E}_{base}$ and $S_{train}$. The blue boxes indicate the predicted bounding boxes, and the red boxes represent the ground truth. Each of the three samples represents different sensor configurations to illustrate the performance across varying scenarios.