Neuromorphic Vision-based Motion Segmentation with Graph Transformer Neural Network

TL;DR

This paper tackles event-based motion segmentation for robotic navigation using neuromorphic vision sensors. It introduces GTNN, a Graph Transformer Neural Network that processes raw event streams as 3D graphs to uncover local and global spatiotemporal correlations without scene priors, and it pairs GTNN with the DOMEL labeling framework to create EMS-DOMEL for benchmarking. The study demonstrates GTNN's superiority over state-of-the-art learning-based and classical methods across public datasets (EV-IMO, MOD, EV-IMO2) and new cross-domain data, achieving substantial gains in IoU and detection rate, while offering improved inference efficiency. The work also provides a scalable training scheme and releases EMS-DOMEL to facilitate further research and benchmarking in event-based motion segmentation, signaling meaningful advances for robust autonomous navigation with neuromorphic sensors.

Abstract

Moving object segmentation is critical to interpret scene dynamics for robotic navigation systems in challenging environments. Neuromorphic vision sensors are tailored for motion perception due to their asynchronous nature, high temporal resolution, and reduced power consumption. However, their unconventional output requires novel perception paradigms to leverage their spatially sparse and temporally dense nature. In this work, we propose a novel event-based motion segmentation algorithm using a Graph Transformer Neural Network, dubbed GTNN. Our proposed algorithm processes event streams as 3D graphs by a series of nonlinear transformations to unveil local and global spatiotemporal correlations between events. Based on these correlations, events belonging to moving objects are segmented from the background without prior knowledge of the dynamic scene geometry. The algorithm is trained on publicly available datasets including MOD, EV-IMO, and \textcolor{black}{EV-IMO2} using the proposed training scheme to facilitate efficient training on extensive datasets. Moreover, we introduce the Dynamic Object Mask-aware Event Labeling (DOMEL) approach for generating approximate ground-truth labels for event-based motion segmentation datasets. We use DOMEL to label our own recorded Event dataset for Motion Segmentation (EMS-DOMEL), which we release to the public for further research and benchmarking. Rigorous experiments are conducted on several unseen publicly-available datasets where the results revealed that GTNN outperforms state-of-the-art methods in the presence of dynamic background variations, motion patterns, and multiple dynamic objects with varying sizes and velocities. GTNN achieves significant performance gains with an average increase of 9.4% and 4.5% in terms of motion segmentation accuracy (IoU%) and detection rate (DR%), respectively.

Figures (11)

• Figure 1: Motion segmentation results of the proposed learning-based algorithm (GTNN) using EV-IMO (Floor sequence) publicly available dataset m6_evimo. (A) APS image for visualization only. (B) Approximate Ground truth events: black represents foreground events and red represents background events. (C) Segmented events using the proposed GTNN algorithm: black indicates predicted foreground events and gray indicates predicted background events. Our GTNN performs a binary classification to differentiate between foreground events due to moving objects or background events due to camera motion.
• Figure 2: Proposed event-based motion segmentation framework based on graph transformer neural network (GTNN). GTNN is developed and trained using the proposed training scheme on publicly available event datasets (EV-IMO, MOD, EV-IMO2) and tested on the corresponding evaluation sequences along with our recorded experiments (EMS-DOMEL). The approximate ground truth event labels of our recorded experiments are generated using the proposed DOMEL approach. The proposed algorithm classifies incoming event streams into foreground events related to moving object(s) or background events.
• Figure 3: Proposed Graph Transformer Neural Network (GTNN) Architecture for Event-based Motion Segmentation. The classification network takes $N$ number of events as input stream, applies input and feature transformations and mapping, and then aggregates global features by global max pooling. The output is a binary classification assigned to each event in the stream, indicating whether it belongs to class 0 (a moving background due to camera motion) or class 1 (dynamic object(s) within the scene).
• Figure 4: Encoding operation of a sample 3-D event graph by GTNN encoder unit, a coupled point transformer layer (stage 1) and a transition down module (stage 2).
• Figure 5: The framework of the proposed effective training scheme compared to the conventional training scheme. Note that $i$ is the current epoch number, $L$ is the number of training subsets, $J=i\%L$ where $\%$ is the modulo operator.