Event Voxel Set Transformer for Spatiotemporal Representation Learning on Event Streams

TL;DR

The paper addresses efficient spatiotemporal representation learning from sparse, asynchronous event streams, a challenge for traditional dense-frame methods. It introduces EVSTr, an Event Voxel Set Transformer that processes voxelized event data through a two-stage encoder: MNEL for robust local aggregation and VSAL for global feature interaction, augmented with absolute-relative positional encoding; a segment modeling strategy S$^{2}$TM enables long-range temporal modeling for action recognition. EVSTr achieves state-of-the-art performance on object classification and action recognition while maintaining low computational cost, demonstrated on datasets including four event-based objects and the NeuroHAR action dataset. The work also provides NeuroHAR, a challenging real-world dataset with low-light and mobile-recording conditions, underscoring the practical impact of sparse-event transformers for real-time recognition and multi-modal potential in future work.

Abstract

Event cameras are neuromorphic vision sensors that record a scene as sparse and asynchronous event streams. Most event-based methods project events into dense frames and process them using conventional vision models, resulting in high computational complexity. A recent trend is to develop point-based networks that achieve efficient event processing by learning sparse representations. However, existing works may lack robust local information aggregators and effective feature interaction operations, thus limiting their modeling capabilities. To this end, we propose an attention-aware model named Event Voxel Set Transformer (EVSTr) for efficient spatiotemporal representation learning on event streams. It first converts the event stream into voxel sets and then hierarchically aggregates voxel features to obtain robust representations. The core of EVSTr is an event voxel transformer encoder that consists of two well-designed components, including the Multi-Scale Neighbor Embedding Layer (MNEL) for local information aggregation and the Voxel Self-Attention Layer (VSAL) for global feature interaction. Enabling the network to incorporate a long-range temporal structure, we introduce a segment modeling strategy (S$^{2}$TM) to learn motion patterns from a sequence of segmented voxel sets. The proposed model is evaluated on two recognition tasks, including object classification and action recognition. To provide a convincing model evaluation, we present a new event-based action recognition dataset (NeuroHAR) recorded in challenging scenarios. Comprehensive experiments show that EVSTr achieves state-of-the-art performance while maintaining low model complexity.

Figures (7)

• Figure 1: (a) Visualization of event camera output. An event camera records the spatiotemporal information of the running action as a stream of events, where red and blue dots denote positive and negative, respectively. (b) Comparison of samples from DVS128 Gesture amir2017dvsgesture and NeuroHAR datasets. The sample is a human action hand clap. After introducing handheld mobile recording, these samples in our proposed NeuroHAR contain both moving subject (zoom box) and background information. Besides, recording in the low-light illumination unlocks the high dynamic range advantage of event cameras.
• Figure 2: (a) Visualization of the human action jumping jacks with three representative event representations. (b) Comparison of different point-based architectures for object classification. Blue, pink and green blocks represent local feature encoding, aggregation function and global modeling, respectively. The competitors include RG-CNNs bi2020rgcnn, EvS-S li2021evs, AEGNN schaefer2022aegnn, EV-VGCNN deng2022evvgcnn, VMV-GCN xie2022vmvgcn, and ECSNet chen2022ecsnet. Our EVSTr has two important improvements: (i) jointly modeling positional and semantic relations between neighbors to achieve multi-scale attentive aggregation; (ii) leveraging self-attention to perform feature interaction in global modeling. $\odot$ denotes Hadamard product and PE is the abbreviation of positional encoding.
• Figure 3: The pipeline of Event Voxel Set Transformer (EVSTr) for object classification and action recognition. We spilt an event stream into $K$ segments of equal duration ($K = 1$ for object classification) and convert them as event voxel sets $\{\mathcal{V}_{1}, \mathcal{V}_{2}, ..., \mathcal{V}_{K}\}$. The event voxel transformer encoder propagates each voxel set to extract spatiotemporal features $\{\hat{\mathbf{A}}_{1}, \hat{\mathbf{A}}_{2}, ..., \hat{\mathbf{A}}_{K}\}$. For object classification, we feed the encoder output into a classification head for predicting classes. For action recognition, we first model long-range temporal dynamics over multiple voxel sets by a Stream-Segment Temporal Modeling Module and then transform the class token into categories. $N_{\rm v}$ represents the total number of input voxels, and $U$ denotes the downsampling rate of voxel sampling.
• Figure 4: The architecture of three components in the EVSTr model. (a) Multi-Scale Neighbor Embedding Layer (MNEL). It attentively aggregates multi-scale neighbor features into a local representation for each voxel. (b) Voxel Self-Attention Layer (VSAL). It performs inter-voxel feature interactions to enhance global representation. We introduce the absolute-relative positional encoding for assisting the self-attention operator in fitting with the spatiotemporal structure. (c) Stream-Segment Temporal Modeling Module (S$^{2}$TM). It is designed for action recognition by learning the long-range temporal dynamics from a sequence of segmented features.
• Figure 5: The t-SNE visualization of feature representations from VMV-GCN xie2022vmvgcn and the proposed EVSTr model on the N-Caltech101 dataset. Best viewed by zooming in.