Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Event Masked Autoencoder: Point-wise Action Recognition with Event-Based Cameras

Jingkai Sun, Qiang Zhang, Jiaxu Wang, Jiahang Cao, Renjing Xu

TL;DR

The paper tackles action recognition from Dynamic Vision Sensor (DVS) event streams, where frame-based representations often discard temporal details. It introduces Event Masked Autoencoder (Event MAE) that treats event streams as 3D point clouds and learns representations by masking patches and reconstructing them, with patch centers chosen by an inlier-based plane fitting and embedded via PointNet for transformer compatibility. A ShapeNet-based pre-training pipeline followed by event-data fine-tuning minimizes Chamfer Distance $CD$ between reconstructed and ground-truth patches, enabling Transformer-friendly embeddings for event cameras. Empirical results show state-of-the-art performance on DVS128-Gesture and SL-Animals-DVS S3 and demonstrate robustness to noise, highlighting the viability of self-supervised, patch-based learning on raw event data and hints at a unified multi-modal backbone for future sensor modalities.

Abstract

Dynamic vision sensors (DVS) are bio-inspired devices that capture visual information in the form of asynchronous events, which encode changes in pixel intensity with high temporal resolution and low latency. These events provide rich motion cues that can be exploited for various computer vision tasks, such as action recognition. However, most existing DVS-based action recognition methods lose temporal information during data transformation or suffer from noise and outliers caused by sensor imperfections or environmental factors. To address these challenges, we propose a novel framework that preserves and exploits the spatiotemporal structure of event data for action recognition. Our framework consists of two main components: 1) a point-wise event masked autoencoder (MAE) that learns a compact and discriminative representation of event patches by reconstructing them from masked raw event camera points data; 2) an improved event points patch generation algorithm that leverages an event data inlier model and point-wise data augmentation techniques to enhance the quality and diversity of event points patches. To the best of our knowledge, our approach introduces the pre-train method into event camera raw points data for the first time, and we propose a novel event points patch embedding to utilize transformer-based models on event cameras.

Event Masked Autoencoder: Point-wise Action Recognition with Event-Based Cameras

TL;DR

The paper tackles action recognition from Dynamic Vision Sensor (DVS) event streams, where frame-based representations often discard temporal details. It introduces Event Masked Autoencoder (Event MAE) that treats event streams as 3D point clouds and learns representations by masking patches and reconstructing them, with patch centers chosen by an inlier-based plane fitting and embedded via PointNet for transformer compatibility. A ShapeNet-based pre-training pipeline followed by event-data fine-tuning minimizes Chamfer Distance between reconstructed and ground-truth patches, enabling Transformer-friendly embeddings for event cameras. Empirical results show state-of-the-art performance on DVS128-Gesture and SL-Animals-DVS S3 and demonstrate robustness to noise, highlighting the viability of self-supervised, patch-based learning on raw event data and hints at a unified multi-modal backbone for future sensor modalities.

Abstract

Dynamic vision sensors (DVS) are bio-inspired devices that capture visual information in the form of asynchronous events, which encode changes in pixel intensity with high temporal resolution and low latency. These events provide rich motion cues that can be exploited for various computer vision tasks, such as action recognition. However, most existing DVS-based action recognition methods lose temporal information during data transformation or suffer from noise and outliers caused by sensor imperfections or environmental factors. To address these challenges, we propose a novel framework that preserves and exploits the spatiotemporal structure of event data for action recognition. Our framework consists of two main components: 1) a point-wise event masked autoencoder (MAE) that learns a compact and discriminative representation of event patches by reconstructing them from masked raw event camera points data; 2) an improved event points patch generation algorithm that leverages an event data inlier model and point-wise data augmentation techniques to enhance the quality and diversity of event points patches. To the best of our knowledge, our approach introduces the pre-train method into event camera raw points data for the first time, and we propose a novel event points patch embedding to utilize transformer-based models on event cameras.
Paper Structure (9 sections, 5 equations, 5 figures, 5 tables)

This paper contains 9 sections, 5 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Event Data Processing
  4. Event Patch Generation For Masked Autoencoder
  5. Event Masked Autoencoder Structure
  6. Experiments
  7. Pre-train Process
  8. Downstream Classification Tasks
  9. Ablation Study

Figures (5)

  • Figure 1: Reconstruction result of right arm rotation clockwise on DVS128-Gestureamir2017low. We present the reconstruction results of a right arm rotation performed clockwise, using the DVS128-Gesture dataset. Specifically, we show the input data, the event data stream obtained after applying the mask, and the resulting reconstruction. In this example, we set the masking ratio to 80$\%$. Compared with the ground truth data, our reconstruction results are denser and less noisy. To visualize the results, we concatenate the 3 event data streams in time. The colors from blue to red indicate the magnitude of the values from 0 to 1 on the x-axis.
  • Figure 2: Pipeline of our Event Masked Autoencoder. In the first half, we present the event patch generation, mask, and embedding process. In the second half, we describe the pre-training process. Due to the asymmetric structure of the encoder and decoder, we differentiate them by their schematic sizes. During the encoding process, only visible tokens are fed into the network, while in the decoding process, masked tokens are added for reconstruction purposes.
  • Figure 3: Reconstruction examples on DVS128-Gesture test set. The colors from blue to red indicate the magnitude of the values from 0 to 1 on the x-axis(normalized).
  • Figure 4: A comparison of three algorithms for generating event patches. We present the ground truth (leftmost), our approach (second from left), random sampling, and FPS (rightmost). Below the figure, we also provide some details about our approach and random sampling. From these details, we can evidently discern that our method has superior performance. The colors from blue to red indicate the magnitude of the values from 0 to 1 on the x-axis.
  • Figure 5: We show three different thresholds. For visualization, we contact two event data streams. When the threshold is 0.35, the connection between the two event data streams is very sparse. When the threshold is 1, the event data stream has more noise.