Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neuromorphic Imaging and Classification with Graph Learning

Pei Zhang, Chutian Wang, Edmund Y. Lam

TL;DR

The paper tackles the challenge of classifying dynamic scenes from asynchronous neuromorphic event streams under limited resources. It introduces an event-to-graph transformation and a Graph Transformer with edge-aware attention (TEA), leveraging temporal degree tau and SAGPool for graph-level classification. Across real and simulated datasets, the approach yields higher accuracy than frame-based and several event-based methods, especially with short event streams, while reducing memory and runtime. This work enables practical mobile and edge deployments of neuromorphic vision systems by efficiently exploiting the temporal and spatial structure of event data.

Abstract

Bio-inspired neuromorphic cameras asynchronously record pixel brightness changes and generate sparse event streams. They can capture dynamic scenes with little motion blur and more details in extreme illumination conditions. Due to the multidimensional address-event structure, most existing vision algorithms cannot properly handle asynchronous event streams. While several event representations and processing methods have been developed to address such an issue, they are typically driven by a large number of events, leading to substantial overheads in runtime and memory. In this paper, we propose a new graph representation of the event data and couple it with a Graph Transformer to perform accurate neuromorphic classification. Extensive experiments show that our approach leads to better results and excels at the challenging realistic situations where only a small number of events and limited computational resources are available, paving the way for neuromorphic applications embedded into mobile facilities.

Neuromorphic Imaging and Classification with Graph Learning

TL;DR

The paper tackles the challenge of classifying dynamic scenes from asynchronous neuromorphic event streams under limited resources. It introduces an event-to-graph transformation and a Graph Transformer with edge-aware attention (TEA), leveraging temporal degree tau and SAGPool for graph-level classification. Across real and simulated datasets, the approach yields higher accuracy than frame-based and several event-based methods, especially with short event streams, while reducing memory and runtime. This work enables practical mobile and edge deployments of neuromorphic vision systems by efficiently exploiting the temporal and spatial structure of event data.

Abstract

Bio-inspired neuromorphic cameras asynchronously record pixel brightness changes and generate sparse event streams. They can capture dynamic scenes with little motion blur and more details in extreme illumination conditions. Due to the multidimensional address-event structure, most existing vision algorithms cannot properly handle asynchronous event streams. While several event representations and processing methods have been developed to address such an issue, they are typically driven by a large number of events, leading to substantial overheads in runtime and memory. In this paper, we propose a new graph representation of the event data and couple it with a Graph Transformer to perform accurate neuromorphic classification. Extensive experiments show that our approach leads to better results and excels at the challenging realistic situations where only a small number of events and limited computational resources are available, paving the way for neuromorphic applications embedded into mobile facilities.
Paper Structure (21 sections, 16 equations, 4 figures, 7 tables)

This paper contains 21 sections, 16 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neuromorphic Camera
  4. Event Learning
  5. Graph Neural Networks
  6. Methodology
  7. Preliminary
  8. Problem Formulation
  9. Event Transformation
  10. Graph Transformer
  11. Experiments
  12. Neuromorphic Data
  13. Comparison on Classification Performance
  14. Classification with Short Event Streams
  15. Comparison of Graph Learning Operators
  16. ...and 6 more sections

Figures (4)

  • Figure 1: The proposed pipeline for classification on events, involving 1 a mapping from events to graphs and 2 a model to learn information from graphs.
  • Figure 2: (a)--(d): Transformations from events into a graph when $t_1 < t_2 < t_3$ and $\mathbf{s}_1 = \mathbf{s}_3$. Color represents polarity. (e): Schematic illustration of the connections of nodes sharing the same position $\mathbf{s}_1 = \mathbf{s}_2 = \mathbf{s}_3$ when $t_1 < t_2 < t_3$. $\mathbf{v}_1$ connects to $\mathbf{v}_2$ and $\mathbf{v}_3$ when $\tau = 2$, while $\mathbf{v}_1$ connects to $\mathbf{v}_2$ only when $\tau = 1$. Best viewed in color and zoom-in.
  • Figure 3: Six kinds of human actions in SN-KTH. Red and blue represent events of different polarity. Best viewed in color.
  • Figure 4: Top-1 classification accuracy as the number of inference events in a batch increases from $10$ to $100$. Best viewed in color and zoom-in.