Event-LAB: Towards Standardized Evaluation of Neuromorphic Localization Methods

TL;DR

The ability of the framework to systematically visualize and analyze the results of multiple methods and datasets is demonstrated, revealing key insights such as the association of parameters that control event collection counts and window sizes for frame generation to large variations in performance.

Abstract

Event-based localization research and datasets are a rapidly growing area of interest, with a tenfold increase in the cumulative total number of published papers on this topic over the past 10 years. Whilst the rapid expansion in the field is exciting, it brings with it an associated challenge: a growth in the variety of required code and package dependencies as well as data formats, making comparisons difficult and cumbersome for researchers to implement reliably. To address this challenge, we present Event-LAB: a new and unified framework for running several event-based localization methodologies across multiple datasets. Event-LAB is implemented using the Pixi package and dependency manager, that enables a single command-line installation and invocation for combinations of localization methods and datasets. To demonstrate the capabilities of the framework, we implement two common event-based localization pipelines: Visual Place Recognition (VPR) and Simultaneous Localization and Mapping (SLAM). We demonstrate the ability of the framework to systematically visualize and analyze the results of multiple methods and datasets, revealing key insights such as the association of parameters that control event collection counts and window sizes for frame generation to large variations in performance. The results and analysis demonstrate the importance of fairly comparing methodologies with consistent event image generation parameters. Our Event-LAB framework provides this ability for the research community, by contributing a streamlined workflow for easily setting up multiple conditions.

Figures (1)

• Figure 1: Overview of the Event-LAB system. Single command-line invocation for a desired baseline method and dataset triggers a series of events. Data is downloaded from an external database and setup to a standard format. Event frames (counts or reconstructions) are generated based on a configuration file consisting of multiple parameters. Baseline methods are cloned from external repositories, including any necessary model weights and checkpoints. Finally, the generated frames and baseline method is run to produce recall and precision metrics for VPR evaluation.