Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EventRPG: Event Data Augmentation with Relevance Propagation Guidance

Mingyuan Sun, Donghao Zhang, Zongyuan Ge, Jiaxu Wang, Jia Li, Zheng Fang, Renjing Xu

TL;DR

This work tackles overfitting in event-based SNN classifiers by deriving saliency maps directly from spiking networks through two propagation rules: Spiking Layer-Time-wise Relevance Propagation (SLTRP) and Spiking Layer-wise Relevance Propagation (SLRP). Building on these, it introduces EventRPG, a relevance-guided augmentation framework with RPGDrop and RPGMix that perturb and merge event streams guided by label-focused saliency, integrated with geometric augmentations. The approach yields high-quality CAMs and saliency maps while delivering state-of-the-art or near-state-of-the-art accuracy on multiple object- and action-recognition benchmarks (e.g., N-Caltech101, CIFAR10-DVS, SL-Animals) with favorable computational efficiency. This combination of interpretable saliency guidance and targeted augmentation substantially improves generalization for event-based vision tasks and demonstrates strong practical impact for real-time, low-power sensing scenarios. Limitations include its current confinement to classification tasks, with future work aiming to extend to multi-task and self-supervised settings.

Abstract

Event camera, a novel bio-inspired vision sensor, has drawn a lot of attention for its low latency, low power consumption, and high dynamic range. Currently, overfitting remains a critical problem in event-based classification tasks for Spiking Neural Network (SNN) due to its relatively weak spatial representation capability. Data augmentation is a simple but efficient method to alleviate overfitting and improve the generalization ability of neural networks, and saliency-based augmentation methods are proven to be effective in the image processing field. However, there is no approach available for extracting saliency maps from SNNs. Therefore, for the first time, we present Spiking Layer-Time-wise Relevance Propagation rule (SLTRP) and Spiking Layer-wise Relevance Propagation rule (SLRP) in order for SNN to generate stable and accurate CAMs and saliency maps. Based on this, we propose EventRPG, which leverages relevance propagation on the spiking neural network for more efficient augmentation. Our proposed method has been evaluated on several SNN structures, achieving state-of-the-art performance in object recognition tasks including N-Caltech101, CIFAR10-DVS, with accuracies of 85.62% and 85.55%, as well as action recognition task SL-Animals with an accuracy of 91.59%. Our code is available at https://github.com/myuansun/EventRPG.

EventRPG: Event Data Augmentation with Relevance Propagation Guidance

TL;DR

This work tackles overfitting in event-based SNN classifiers by deriving saliency maps directly from spiking networks through two propagation rules: Spiking Layer-Time-wise Relevance Propagation (SLTRP) and Spiking Layer-wise Relevance Propagation (SLRP). Building on these, it introduces EventRPG, a relevance-guided augmentation framework with RPGDrop and RPGMix that perturb and merge event streams guided by label-focused saliency, integrated with geometric augmentations. The approach yields high-quality CAMs and saliency maps while delivering state-of-the-art or near-state-of-the-art accuracy on multiple object- and action-recognition benchmarks (e.g., N-Caltech101, CIFAR10-DVS, SL-Animals) with favorable computational efficiency. This combination of interpretable saliency guidance and targeted augmentation substantially improves generalization for event-based vision tasks and demonstrates strong practical impact for real-time, low-power sensing scenarios. Limitations include its current confinement to classification tasks, with future work aiming to extend to multi-task and self-supervised settings.

Abstract

Event camera, a novel bio-inspired vision sensor, has drawn a lot of attention for its low latency, low power consumption, and high dynamic range. Currently, overfitting remains a critical problem in event-based classification tasks for Spiking Neural Network (SNN) due to its relatively weak spatial representation capability. Data augmentation is a simple but efficient method to alleviate overfitting and improve the generalization ability of neural networks, and saliency-based augmentation methods are proven to be effective in the image processing field. However, there is no approach available for extracting saliency maps from SNNs. Therefore, for the first time, we present Spiking Layer-Time-wise Relevance Propagation rule (SLTRP) and Spiking Layer-wise Relevance Propagation rule (SLRP) in order for SNN to generate stable and accurate CAMs and saliency maps. Based on this, we propose EventRPG, which leverages relevance propagation on the spiking neural network for more efficient augmentation. Our proposed method has been evaluated on several SNN structures, achieving state-of-the-art performance in object recognition tasks including N-Caltech101, CIFAR10-DVS, with accuracies of 85.62% and 85.55%, as well as action recognition task SL-Animals with an accuracy of 91.59%. Our code is available at https://github.com/myuansun/EventRPG.
Paper Structure (46 sections, 1 theorem, 18 equations, 9 figures, 7 tables)

This paper contains 46 sections, 1 theorem, 18 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Spiking Relevance Propagation Rule
  4. Relevance Propagation for Linear Layers in SNNs
  5. Spiking Layer-Time-wise Relevance Propagation
  6. Spiking Layer-wise Relevance Propagation
  7. Relevance Propagation Guided Event Data Augmentation
  8. Saliency Map and Class Activation Map
  9. Relevance Propagation Guided Event Drop
  10. Relevance Propagation Guided Event Mix
  11. EventRPG
  12. Experiments
  13. Effectiveness of SLRP and SLTRP
  14. Objective Faithfulness
  15. Evaluation of Selectivity
  16. ...and 31 more sections

Key Result

Proposition 1

The sum of relevance scores propagated after spiking layer $l$ from time step $1$ to $k (k<T)$ is

Figures (9)

  • Figure 1: Forward propagation flow and relevance propagation flow of ANNs (a) and SNNs (b, c).
  • Figure 2: Illustration of Relevance Propagation Guided Event Mix and Event Drop. (a) RPGDrop. Where the saliency map offers a higher value, events are more likely to be dropped.(b) RPGMix.
  • Figure 3: Augmentation process of EventRPG.
  • Figure 4: CAM and saliency map by different methods on (a) N-Caltech101 and (b) CIFAR10-DVS. (c) Saliency maps generated from SLTRP on DVS-Gesture and SL-Animals.
  • Figure 5: Average time cost of augmentation methods with SEW Resnet18 on two datasets.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Definition 1: Conservation Property
  • Proposition 1
  • proof
  • proof