Asynchronous Blob Tracker for Event Cameras

TL;DR

A novel algorithm for tracking event blobs using raw events asynchronously in real time using a nearest neighbor classifier with a dynamic threshold criteria for data association coupled with an extended Kalman filter to track the event blob state.

Abstract

Event-based cameras are popular for tracking fast-moving objects due to their high temporal resolution, low latency, and high dynamic range. In this paper, we propose a novel algorithm for tracking event blobs using raw events asynchronously in real time. We introduce the concept of an event blob as a spatio-temporal likelihood of event occurrence where the conditional spatial likelihood is blob-like. Many real-world objects such as car headlights or any quickly moving foreground objects generate event blob data. The proposed algorithm uses a nearest neighbour classifier with a dynamic threshold criteria for data association coupled with an extended Kalman filter to track the event blob state. Our algorithm achieves highly accurate blob tracking, velocity estimation, and shape estimation even under challenging lighting conditions and high-speed motions (> 11000 pixels/s). The microsecond time resolution achieved means that the filter output can be used to derive secondary information such as time-to-contact or range estimation, that will enable applications to real-world problems such as collision avoidance in autonomous driving.

Figures (16)

• Figure 1: Our asynchronous event blob tracker leverages asynchronous raw events (top row) to provide high-bandwidth estimates of an event blob's position ($p$), linear ($v$) and angular ($q$) velocity, orientation ($\theta$), and shape ($\lambda$). The filter operates effectively in challenging scenarios including high-speed motions and extreme lighting conditions, and yields state updates up to microsecond resolution depending on event rates. Potential applications include tracking multi-vehicle tail lights (left bottom) or aerial vehicles (right bottom). The input event data are shown in the first row. The high data rate trajectory estimated by our algorithm is marked by red lines and they traverse through the 40Hz reference images along the time axis. Note that images are blind between frames and are only used for visualisation. The high temporal resolution and shape estimation provided by the filter enable downstream data processing such as time-to-contact (TTC) and range estimation.
• Figure 2: An example of plotting a short temporal window of events in a histogram pseudo-frame for the two case studies considered in the paper. In the upper row, the LED car tail lights create a collection of event blobs. (Note the reflections of the lights in the bonnet of the experimental car.) In the bottom row, a quadrotor is flying in front of the trees and is difficult to pick out of the RGB image but generates an event blob that is clearly visible in (e) and (f). The leading and trailing edges of the event blob can be seen in (e), while the overall blob structure is visible in (f).
• Figure 3: Block diagram of our asynchronous event blob tracking algorithm. The asynchronous filter state $\hat{x}_k$ updates with each event associated to a specific blob. Using the proposed two-stage measurement functions, the Extended Kalman filter (EKF) iteratively generates the optimal estimate for the state. The first function measures the normalised position error of the state, and the second function measures the variance of the chi-squared distribution formulated by the recent events and states saved in the state estimation buffer. Predicting camera ego-motion using gyroscope angular velocity measurements is optional in the EKF and can be included when a calibrated IMU is available.
• Figure 4: Example of modelling the polarity offset vector $\Delta$ in the filter state to compensate the bimodal distribution of event data for tracking a non-flickering moving object. Left: an intensity frame showing a black target spinning clockwise. Middle: corresponding event data showing a bimodal distribution where the leading edge events are mostly negative (blue) and the trailing edge events are positive (red). Right: event data distribution compensated by our polarity offset vector $\Delta$. The estimated velocity $v$ (black arrow), the polarity offset vector $\Delta$ (green arrow) and the estimated shape $\lambda$ (orange ellipse) are shown in the zoom in the right top corner of the right-most figure.
• Figure 5: Example of evaluating the asynchronous event-based tracking methods ACE Alzugaray18threedv, HASTE alzugaray2020haste, Prophesee Prophesee, jAER jAER and our AEB tracker. The target moves in a circular trajectory, increasing speed from $\sim$100 pixels/s to more than 11000 pixels/s over 90 seconds. The main figure illustrates the $x$ position estimate generated by each algorithm. The lines above this figure show the speed range in which each algorithm operates robustly without losing track. Dotted continuation of these lines indicates that the algorithm is able to reinitialise but continues to fail with progressively shorter life times. The solid dot terminations indicate the end-of-life of the target track while the open circle terminations to the dotted line indicates that future tracks last less than one cycle. The sub-figures (a)-(c) provide zoomed-in detail in short time windows around the key changes in behaviour. Figure (a) shows the ACE Alzugaray18threedv and the four HASTE algorithms alzugaray2020haste losing track. These algorithms initially can recapture track as the target circles and recrosses the current state estimate. Figure (b) shows the Prophesee Prophesee method losing track. The Prophesee Prophesee deletes track hypothesis that no longer correspond to an active track (indicated by the solid dot termination in the subplot). A new ID could be selected to continue tracking, however, without prior knowledge about which of the possible targets correspond to the desired blob such a process cannot be considered a natural part of the tracking algorithm. Figure (c) demonstrates the jAER jAER method losing target track. Once again the algorithm provides a collection of alternative hypothesis for targets to track each of which can be tracked for a short time, however, the initial loss of track indicates the speed at which the algorithm becomes unreliable.