TOFFE -- Temporally-binned Object Flow from Events for High-speed and Energy-Efficient Object Detection and Tracking

TL;DR

TOFFE addresses the need for fast, energy-efficient object motion estimation on resource-constrained edge devices byIntroducing a temporally-binned event-based pipeline that merges Spiking Neural Networks with conventional ANNs. It decomposes Object Flow into speed separation (OFS) using a trainable SNN and pose/direction estimation (OFPD) using a lightweight CNN, operating on a discretized event-volume. A synthetic TOFFE dataset enables supervised training for high-speed motion scenarios, outperforming prior lightweight baselines in energy and latency, especially on hybrid neuromorphic hardware. The approach offers robust, real-time capable perception for autonomous navigation in high-speed contexts, with practical implications for drones and small robots deploying event-based vision.

Abstract

Object detection and tracking is an essential perception task for enabling fully autonomous navigation in robotic systems. Edge robot systems such as small drones need to execute complex maneuvers at high-speeds with limited resources, which places strict constraints on the underlying algorithms and hardware. Traditionally, frame-based cameras are used for vision-based perception due to their rich spatial information and simplified synchronous sensing capabilities. However, obtaining detailed information across frames incurs high energy consumption and may not even be required. In addition, their low temporal resolution renders them ineffective in high-speed motion scenarios. Event-based cameras offer a biologically-inspired solution to this by capturing only changes in intensity levels at exceptionally high temporal resolution and low power consumption, making them ideal for high-speed motion scenarios. However, their asynchronous and sparse outputs are not natively suitable with conventional deep learning methods. In this work, we propose TOFFE, a lightweight hybrid framework for performing event-based object motion estimation (including pose, direction, and speed estimation), referred to as Object Flow. TOFFE integrates bio-inspired Spiking Neural Networks (SNNs) and conventional Analog Neural Networks (ANNs), to efficiently process events at high temporal resolutions while being simple to train. Additionally, we present a novel event-based synthetic dataset involving high-speed object motion to train TOFFE. Our experimental results show that TOFFE achieves 5.7x/8.3x reduction in energy consumption and 4.6x/5.8x reduction in latency on edge GPU(Jetson TX2)/hybrid hardware(Loihi-2 and Jetson TX2), compared to previous event-based object detection baselines.

Figures (6)

• Figure 1: a) Color wheels for optical flow and object flow. Optical flow has a continuous speed representation while object flow uses a discretized speed representation with arrows depicting the motion direction. b)The raw stream of events in a given time window is discretized into five event-bins. These bins represent inputs at different timesteps when passed sequentially to an SNN. c) Comparison of TOFFE with Adaptive-spikenet adaptivespikenet and DOTIE dotie for object flow estimation.
• Figure 2: a) A Leaky-Integrate and Fire (LIF) spiking neuron with firing threshold ($v^{th}_l$) and leak factor ($\lambda$). b) SNN-based OFS (Object Flow Speed) block. c) ANN-based OFPD (Object Flow Pose and Direction) block.
• Figure 3: a) Train and Test set trajectories for for TOFFE dataset. b) Accumulated event-bin images corresponding to speeds - 1 to 4 for lemnniscate trajectory rendered at 30FPS.
• Figure 4: Data collection setup in Gazebo.
• Figure 5: TOFFE Training with four speeds (N=4). TOFFE is trained on many different objects, trajectories and speeds. TOFFE learns the pose, direction and speed of objects in the image plane.