Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty-aware Bridge based Mobile-Former Network for Event-based Pattern Recognition

Haoxiang Yang, Chengguo Yuan, Yabin Zhu, Lan Chen, Xiao Wang, Futian Wang

TL;DR

This paper addresses the limitations of RGB-based HAR in challenging conditions and privacy concerns by leveraging event cameras. It proposes a lightweight uncertainty-aware Mobile-Former network that fuses CNN and Transformer branches through a Gaussian-based UA-Bridge to enable effective local-global feature learning for event streams. The authors demonstrate state-of-the-art or strong results across multiple event-based benchmarks, supported by extensive ablations showing the benefits of the uncertainty-driven information propagation, cross-attention fusion, and dynamic ReLU. The approach offers efficient, accurate, and privacy-friendly pattern recognition for event streams, with potential for further gains via distillation in future work.

Abstract

The mainstream human activity recognition (HAR) algorithms are developed based on RGB cameras, which are easily influenced by low-quality images (e.g., low illumination, motion blur). Meanwhile, the privacy protection issue caused by ultra-high definition (HD) RGB cameras aroused more and more people's attention. Inspired by the success of event cameras which perform better on high dynamic range, no motion blur, and low energy consumption, we propose to recognize human actions based on the event stream. We propose a lightweight uncertainty-aware information propagation based Mobile-Former network for efficient pattern recognition, which aggregates the MobileNet and Transformer network effectively. Specifically, we first embed the event images using a stem network into feature representations, then, feed them into uncertainty-aware Mobile-Former blocks for local and global feature learning and fusion. Finally, the features from MobileNet and Transformer branches are concatenated for pattern recognition. Extensive experiments on multiple event-based recognition datasets fully validated the effectiveness of our model. The source code of this work will be released at https://github.com/Event-AHU/Uncertainty_aware_MobileFormer.

Uncertainty-aware Bridge based Mobile-Former Network for Event-based Pattern Recognition

TL;DR

This paper addresses the limitations of RGB-based HAR in challenging conditions and privacy concerns by leveraging event cameras. It proposes a lightweight uncertainty-aware Mobile-Former network that fuses CNN and Transformer branches through a Gaussian-based UA-Bridge to enable effective local-global feature learning for event streams. The authors demonstrate state-of-the-art or strong results across multiple event-based benchmarks, supported by extensive ablations showing the benefits of the uncertainty-driven information propagation, cross-attention fusion, and dynamic ReLU. The approach offers efficient, accurate, and privacy-friendly pattern recognition for event streams, with potential for further gains via distillation in future work.

Abstract

The mainstream human activity recognition (HAR) algorithms are developed based on RGB cameras, which are easily influenced by low-quality images (e.g., low illumination, motion blur). Meanwhile, the privacy protection issue caused by ultra-high definition (HD) RGB cameras aroused more and more people's attention. Inspired by the success of event cameras which perform better on high dynamic range, no motion blur, and low energy consumption, we propose to recognize human actions based on the event stream. We propose a lightweight uncertainty-aware information propagation based Mobile-Former network for efficient pattern recognition, which aggregates the MobileNet and Transformer network effectively. Specifically, we first embed the event images using a stem network into feature representations, then, feed them into uncertainty-aware Mobile-Former blocks for local and global feature learning and fusion. Finally, the features from MobileNet and Transformer branches are concatenated for pattern recognition. Extensive experiments on multiple event-based recognition datasets fully validated the effectiveness of our model. The source code of this work will be released at https://github.com/Event-AHU/Uncertainty_aware_MobileFormer.
Paper Structure (15 sections, 4 equations, 5 figures, 4 tables)

This paper contains 15 sections, 4 equations, 5 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Event-based Recognition
  4. Uncertainty-aware Learning
  5. METHODOLOGY
  6. Overview
  7. Network Architecture
  8. EXPERIMENTS
  9. Dataset and Evaluation Metric
  10. Implementation Details
  11. Comparison with Other SOTA Algorithms
  12. Ablation Study
  13. Parameter Analysis
  14. Visualization
  15. Conclusion

Figures (5)

  • Figure 1: An overview of our proposed uncertainty-aware bridge based Mobile-Former framework for event-based pattern recognition.
  • Figure 2: Ablation study on parameter analysis.
  • Figure 3: Visualization of the CNN features using our model on the N-Caltech101 dataset.
  • Figure 4: Visualization of the top-5 predicted categories using our model on the N-Caltech101 dataset.
  • Figure 5: Visualization of our feature distribution on N-Caltech101.