Self-Supervised Event Representations: Towards Accurate, Real-Time Perception on SoC FPGAs

TL;DR

This work tackles the challenge of real-time, accurate perception from sparse, asynchronous event-camera streams by introducing Self-Supervised Event Representation (SSER), a GRU-based per-pixel encoding that preserves exact event timestamps and polarities without discretising time. The method builds a latent memory $H \in \mathbb{R}^{W\times H\times C}$ updated event-by-event, with a self-supervised encoder–decoder objective that enables rapid, asynchronous inference and strong object-detection performance. It demonstrates quantitative gains over handcrafted aggregation baselines on Gen1 and 1 Mpx datasets (2.4% and 0.6% mAP improvements, respectively) and presents the first hardware implementation of a recurrent event-representation generator on an SoC FPGA, achieving sub-microsecond latency and a 1–2 W power envelope. The work highlights the value of recurrence for maintaining temporal richness in dense representations and paves the way for energy-efficient, high-throughput event-based perception in embedded systems.

Abstract

Event cameras offer significant advantages over traditional frame-based sensors. These include microsecond temporal resolution, robustness under varying lighting conditions and low power consumption. Nevertheless, the effective processing of their sparse, asynchronous event streams remains challenging. Existing approaches to this problem can be categorised into two distinct groups. The first group involves the direct processing of event data with neural models, such as Spiking Neural Networks or Graph Convolutional Neural Networks. However, this approach is often accompanied by a compromise in terms of qualitative performance. The second group involves the conversion of events into dense representations with handcrafted aggregation functions, which can boost accuracy at the cost of temporal fidelity. This paper introduces a novel Self-Supervised Event Representation (SSER) method leveraging Gated Recurrent Unit (GRU) networks to achieve precise per-pixel encoding of event timestamps and polarities without temporal discretisation. The recurrent layers are trained in a self-supervised manner to maximise the fidelity of event-time encoding. The inference is performed with event representations generated asynchronously, thus ensuring compatibility with high-throughput sensors. The experimental validation demonstrates that SSER outperforms aggregation-based baselines, achieving improvements of 2.4% mAP and 0.6% on the Gen1 and 1 Mpx object detection datasets. Furthermore, the paper presents the first hardware implementation of recurrent representation for event data on a System-on-Chip FPGA, achieving sub-microsecond latency and power consumption between 1-2 W, suitable for real-time, power-efficient applications. Code is available at https://github.com/vision-agh/RecRepEvent.

Figures (4)

• Figure 1: Overview of the proposed self‑supervised event‑representation method. Sparse spatio‑temporal events (a) are processed one by one by a stack of pretrained gated‑recurrent‑unit layers (b), which update a latent‑memory tensor at the corresponding pixel $(x,y)$. The latent memory has the same spatial resolution as the input and is updated continuously throughout the time window. After all events have been processed, the final latent memory forms a dense representation (c) that preserves temporal information $t$ and polarity $p$.
• Figure 2: Overview of the training procedure for the proposed event encoder. Blue and red colours represent polarity, while shading indicates time. The spatio-temporal sequence of events (a) is first transformed into a tensor (b), where length $Z$ corresponds to the maximum number of events per pixel in the sequence. The tensor is then linearised (c), and the encoder iterates over the $Z$ timestamps, resulting in a $WH \times C$ representation (d). After this, the decoder reconstructs the events in a rolling manner based on the encoded representation (e). The loss is calculated between the input and the reconstructed events.
• Figure 3: Ablation studies over output sizes and quantisation bit resolution for encoders.
• Figure 4: Example visualization of 12 GRU encoder channels on a sample from a 1-megapixel dataset. Values range from -1 to 1, with blue indicating negative values and red indicating positive values.