Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SMV-EAR: Bring Spatiotemporal Multi-View Representation Learning into Efficient Event-Based Action Recognition

Rui Fan, Weidong Hao

TL;DR

This work tackles the problem of robust spatiotemporal modeling in event-based action recognition by addressing translation-variant representations and naive fusion in existing SMVRL methods. It proposes a threefold framework: Translation-Invariant Spatiotemporal Multi-view (TISM) representations, a Dual-branch Dynamic Cross-view Fusion (DDCF) architecture for sample-wise view integration, and a Diverse Temporal Warping (DTW) data augmentation to simulate speed variability. Empirically, SMV-EAR delivers state-of-the-art Top-1 accuracy on HARDVS, DailyDVS-200, and THU-EACT-50-CHL with substantial reductions in parameters and MACs, validating the effectiveness and efficiency of translation-invariant multi-view learning for EAR. The contributions offer a principled, scalable SMVRL paradigm that enhances temporal modeling and cross-view collaboration in event-based vision with potential applicability to broader EAR tasks.

Abstract

Event cameras action recognition (EAR) offers compelling privacy-protecting and efficiency advantages, where temporal motion dynamics is of great importance. Existing spatiotemporal multi-view representation learning (SMVRL) methods for event-based object recognition (EOR) offer promising solutions by projecting H-W-T events along spatial axis H and W, yet are limited by its translation-variant spatial binning representation and naive early concatenation fusion architecture. This paper reexamines the key SMVRL design stages for EAR and propose: (i) a principled spatiotemporal multi-view representation through translation-invariant dense conversion of sparse events, (ii) a dual-branch, dynamic fusion architecture that models sample-wise complementarity between motion features from different views, and (iii) a bio-inspired temporal warping augmentation that mimics speed variability of real-world human actions. On three challenging EAR datasets of HARDVS, DailyDVS-200 and THU-EACT-50-CHL, we show +7.0%, +10.7%, and +10.2% Top-1 accuracy gains over existing SMVRL EOR method with surprising 30.1% reduced parameters and 35.7% lower computations, establishing our framework as a novel and powerful EAR paradigm.

SMV-EAR: Bring Spatiotemporal Multi-View Representation Learning into Efficient Event-Based Action Recognition

TL;DR

This work tackles the problem of robust spatiotemporal modeling in event-based action recognition by addressing translation-variant representations and naive fusion in existing SMVRL methods. It proposes a threefold framework: Translation-Invariant Spatiotemporal Multi-view (TISM) representations, a Dual-branch Dynamic Cross-view Fusion (DDCF) architecture for sample-wise view integration, and a Diverse Temporal Warping (DTW) data augmentation to simulate speed variability. Empirically, SMV-EAR delivers state-of-the-art Top-1 accuracy on HARDVS, DailyDVS-200, and THU-EACT-50-CHL with substantial reductions in parameters and MACs, validating the effectiveness and efficiency of translation-invariant multi-view learning for EAR. The contributions offer a principled, scalable SMVRL paradigm that enhances temporal modeling and cross-view collaboration in event-based vision with potential applicability to broader EAR tasks.

Abstract

Event cameras action recognition (EAR) offers compelling privacy-protecting and efficiency advantages, where temporal motion dynamics is of great importance. Existing spatiotemporal multi-view representation learning (SMVRL) methods for event-based object recognition (EOR) offer promising solutions by projecting H-W-T events along spatial axis H and W, yet are limited by its translation-variant spatial binning representation and naive early concatenation fusion architecture. This paper reexamines the key SMVRL design stages for EAR and propose: (i) a principled spatiotemporal multi-view representation through translation-invariant dense conversion of sparse events, (ii) a dual-branch, dynamic fusion architecture that models sample-wise complementarity between motion features from different views, and (iii) a bio-inspired temporal warping augmentation that mimics speed variability of real-world human actions. On three challenging EAR datasets of HARDVS, DailyDVS-200 and THU-EACT-50-CHL, we show +7.0%, +10.7%, and +10.2% Top-1 accuracy gains over existing SMVRL EOR method with surprising 30.1% reduced parameters and 35.7% lower computations, establishing our framework as a novel and powerful EAR paradigm.
Paper Structure (17 sections, 7 equations, 11 figures, 8 tables, 1 algorithm)

This paper contains 17 sections, 7 equations, 11 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Event-Based Action Recognition
  4. SMVRL
  5. Data Augmentation for Event Data
  6. Methodology
  7. Overall Framework
  8. Translation-invariance of TISM
  9. Dual-branch and Dynamic DDCF
  10. Diverse Speed Variations of DTW
  11. Experiments
  12. Experimental Setup
  13. Comparison with the State-of-the-Arts
  14. Ablation Studies
  15. Visualization and Analysis.
  16. ...and 2 more sections

Figures (11)

  • Figure 1: Comparison of frame-like representation in prior SOTA EAR method (left) and our adopted multi-view representation (right). Our SMV-EAR embeds motion cues within $T$-$H$ and $T$-$W$ maps rather than between $H$-$W$ frames.
  • Figure 2: Projections of an action sample on different views.
  • Figure 3: Comparisons of representation and architecture between baseline SMVRL method deng2021mvf (top) and our SMV-EAR (bottom). SMV-EAR ensures translation-invariant and reasonable SMVRL.
  • Figure 4: Results on HARDVS dataset. Our SMV-EAR surpasses all methods and sets a new performance frontier for EAR task.
  • Figure 5: Overview of main contributions in SMV-EAR's pipeline.
  • ...and 6 more figures