Asynchronous Multi-Object Tracking with an Event Camera

TL;DR

The paper introduces AEMOT, an asynchronous multi-object tracking framework for event cameras that combines an AEB-based tracker with a lightweight validation stage. Detection is based on a Field of Active Flow Directions derived from SAE, enabling per-event blob candidates to be spawned and validated via intensity patches. AEB-based tracking, EKF fusion, and data association support robust, high-speed tracking of dozens of small objects (bees) in a challenging bee swarm dataset, outperforming state-of-the-art event-based trackers in precision and recall. The work provides a publicly releasable Bee Swarm Dataset and code, and the validation network can be retrained for other object classes, enabling broader applicability of asynchronous event-based tracking.

Abstract

Events cameras are ideal sensors for enabling robots to detect and track objects in highly dynamic environments due to their low latency output, high temporal resolution, and high dynamic range. In this paper, we present the Asynchronous Event Multi-Object Tracking (AEMOT) algorithm for detecting and tracking multiple objects by processing individual raw events asynchronously. AEMOT detects salient event blob features by identifying regions of consistent optical flow using a novel Field of Active Flow Directions built from the Surface of Active Events. Detected features are tracked as candidate objects using the recently proposed Asynchronous Event Blob (AEB) tracker in order to construct small intensity patches of each candidate object. A novel learnt validation stage promotes or discards candidate objects based on classification of their intensity patches, with promoted objects having their position, velocity, size, and orientation estimated at their event rate. We evaluate AEMOT on a new Bee Swarm Dataset, where it tracks dozens of small bees with precision and recall performance exceeding that of alternative event-based detection and tracking algorithms by over 37%. Source code and the labelled event Bee Swarm Dataset will be open sourced

Figures (5)

• Figure 1: Our Asynchronous Event Multi-Object Tracking (AEMOT) algorithm effectively detects and tracks dozens of bees in the Bee Swarm Dateset using an asynchronous detection, validation, and tracking approach. This approach provides robustness against background activity and enables the detection, validation, and tracking of multiple objects at the temporal resolution of their individual event rates
• Figure 2: Example intensity patches used for track validation on the Bee Swarm Dataset, where the intensity of events is shown. (a) shows true tracks (bees) where the leading and trailing are clear and there is a well-defined structure. (b) shows false tracks (background) where this structure is not seen and there is a more random distribution of events.
• Figure 3: Block diagram of AEMOT. For each event, data association is performed (in priority order) to assign the event to any existing valid or candidate tracks, otherwise detection is performed. All tracks are evaluated using position and time since the last associated event. The classification network is evaluated periodically, which can be configured to suit the application. Following evaluation, valid tracks are deleted or continue, and candidate tracks are promoted, deleted or continue.
• Figure 4: Sample frames (with tracks) at $t=1.13$s for (a) AEMOT, (b) Prophesee 2024_Prophesee_spatter_tracker, and (c) jAER 2008_Delbruck_jAER. Importantly, this frame-like representation is not used in AEMOT as each event is processed asynchronously. See a detailed comparison in our supplementary video.
• Figure 5: Example of crossing paths in Bee Swarm Dataset, where three bees pass within close proximity. The pre-associated events (a) are shown with colour indicating the event time. The same events are shown after data association (b), with the solid lines indicating the position estimate of the respective tracks. Note that an event can be associated to multiple tracks.