Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Maximizing Asynchronicity in Event-based Neural Networks

Haiqing Hao, Nikola Zubić, Weihua He, Zhipeng Sui, Davide Scaramuzza, Wenhui Wang

TL;DR

The paper addresses the challenge of applying machine learning to high-temporal-resolution, asynchronous event camera data by introducing EVA, an asynchronous-to-synchronous framework. EVA uses a linear-attention–based encoder with matrix-valued hidden states (MVHS) and patch-wise encoding, coupled with self-supervised learning via multi-representation prediction and next representation prediction to learn generalizable representations. It achieves state-of-the-art results on event-based recognition benchmarks and becomes the first A2S method to excel at object detection on Gen1, demonstrating strong potential for real-time, event-based vision. The approach provides a practical pathway to leverage event sparsity and temporal precision in real-time systems while maintaining compatibility with standard ML pipelines.

Abstract

Event cameras deliver visual data with high temporal resolution, low latency, and minimal redundancy, yet their asynchronous, sparse sequential nature challenges standard tensor-based machine learning (ML). While the recent asynchronous-to-synchronous (A2S) paradigm aims to bridge this gap by asynchronously encoding events into learned representations for ML pipelines, existing A2S approaches often sacrifice representation expressivity and generalizability compared to dense, synchronous methods. This paper introduces EVA (EVent Asynchronous representation learning), a novel A2S framework to generate highly expressive and generalizable event-by-event representations. Inspired by the analogy between events and language, EVA uniquely adapts advances from language modeling in linear attention and self-supervised learning for its construction. In demonstration, EVA outperforms prior A2S methods on recognition tasks (DVS128-Gesture and N-Cars), and represents the first A2S framework to successfully master demanding detection tasks, achieving a remarkable 47.7 mAP on the Gen1 dataset. These results underscore EVA's transformative potential for advancing real-time event-based vision applications.

Maximizing Asynchronicity in Event-based Neural Networks

TL;DR

The paper addresses the challenge of applying machine learning to high-temporal-resolution, asynchronous event camera data by introducing EVA, an asynchronous-to-synchronous framework. EVA uses a linear-attention–based encoder with matrix-valued hidden states (MVHS) and patch-wise encoding, coupled with self-supervised learning via multi-representation prediction and next representation prediction to learn generalizable representations. It achieves state-of-the-art results on event-based recognition benchmarks and becomes the first A2S method to excel at object detection on Gen1, demonstrating strong potential for real-time, event-based vision. The approach provides a practical pathway to leverage event sparsity and temporal precision in real-time systems while maintaining compatibility with standard ML pipelines.

Abstract

Event cameras deliver visual data with high temporal resolution, low latency, and minimal redundancy, yet their asynchronous, sparse sequential nature challenges standard tensor-based machine learning (ML). While the recent asynchronous-to-synchronous (A2S) paradigm aims to bridge this gap by asynchronously encoding events into learned representations for ML pipelines, existing A2S approaches often sacrifice representation expressivity and generalizability compared to dense, synchronous methods. This paper introduces EVA (EVent Asynchronous representation learning), a novel A2S framework to generate highly expressive and generalizable event-by-event representations. Inspired by the analogy between events and language, EVA uniquely adapts advances from language modeling in linear attention and self-supervised learning for its construction. In demonstration, EVA outperforms prior A2S methods on recognition tasks (DVS128-Gesture and N-Cars), and represents the first A2S framework to successfully master demanding detection tasks, achieving a remarkable 47.7 mAP on the Gen1 dataset. These results underscore EVA's transformative potential for advancing real-time event-based vision applications.
Paper Structure (25 sections, 15 equations, 5 figures, 9 tables)

This paper contains 25 sections, 15 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Asynchronous event processing
  4. Event data-based self-supervised representation learning
  5. Method
  6. Architecture: asynchronous encoder
  7. Event tokenization
  8. Embedding events with varied temporal intervals
  9. Modeling event sequence with linear attention
  10. Patch-wise representation encoding
  11. Self-supervised representation learning on asynchronous events
  12. Multi-representation prediction
  13. Next representation prediction
  14. Experiments
  15. Object recognition
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Overview of the proposed EVA A2S framework. It features an asynchronous LA-based encoder (top middle) producing event-by-event updated representations for synchronous downstream tasks (right). These representations are learned via a self-supervised module (bottom middle) and are event-by-event updated during inference to be sampled on-demand by downstream tasks.
  • Figure 2: Parallels between event data and language: their sequential nature and incremental manner.
  • Figure 3: Architecture of the A2S framework: (a) Tokenizing and embedding events. (b) Asynchronous encoding via LA. (c) MRP and NRP for self-supervised representation learning.
  • Figure 4: Patch-wise representation encoding. Events are partitioned and encoded by patch. Patch representations could be concatenated for downstream tasks.
  • Figure 5: Visualization on DVS128-Gesture. (a) Ground truth of the handcrafted representations. (b) Predicted results of the handcrafted representations. (c) The learned representations.