Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Predictive Temporal Attention on Event-based Video Stream for Energy-efficient Situation Awareness

Yiming Bu, Jiayang Liu, Qinru Qiu

TL;DR

The paper tackles energy efficiency in DVS-based vision by gating camera output through predictive temporal attention. It introduces an SNN-ANN hybrid autoencoder predictor paired with an evaluator-based gating mechanism, and formalizes an Event Similarity Esim metric (Esim(F1,F2) = |F1 ∩ F2| / |F1 ∪ F2|) to quantify prediction quality; Region Esim extends this to tolerate noise and shifts. Empirically, the approach reduces data communication by 46.7% and computation by 43.8% while maintaining situation awareness, with the predictor effectively filtering noise and the evaluator-guided gating adapting to prediction quality. The method is validated across multiple datasets, demonstrating energy savings and robustness in event-based perception systems.

Abstract

The Dynamic Vision Sensor (DVS) is an innovative technology that efficiently captures and encodes visual information in an event-driven manner. By combining it with event-driven neuromorphic processing, the sparsity in DVS camera output can result in high energy efficiency. However, similar to many embedded systems, the off-chip communication between the camera and processor presents a bottleneck in terms of power consumption. Inspired by the predictive coding model and expectation suppression phenomenon found in human brain, we propose a temporal attention mechanism to throttle the camera output and pay attention to it only when the visual events cannot be well predicted. The predictive attention not only reduces power consumption in the sensor-processor interface but also effectively decreases the computational workload by filtering out noisy events. We demonstrate that the predictive attention can reduce 46.7% of data communication between the camera and the processor and reduce 43.8% computation activities in the processor.

Predictive Temporal Attention on Event-based Video Stream for Energy-efficient Situation Awareness

TL;DR

The paper tackles energy efficiency in DVS-based vision by gating camera output through predictive temporal attention. It introduces an SNN-ANN hybrid autoencoder predictor paired with an evaluator-based gating mechanism, and formalizes an Event Similarity Esim metric (Esim(F1,F2) = |F1 ∩ F2| / |F1 ∪ F2|) to quantify prediction quality; Region Esim extends this to tolerate noise and shifts. Empirically, the approach reduces data communication by 46.7% and computation by 43.8% while maintaining situation awareness, with the predictor effectively filtering noise and the evaluator-guided gating adapting to prediction quality. The method is validated across multiple datasets, demonstrating energy savings and robustness in event-based perception systems.

Abstract

The Dynamic Vision Sensor (DVS) is an innovative technology that efficiently captures and encodes visual information in an event-driven manner. By combining it with event-driven neuromorphic processing, the sparsity in DVS camera output can result in high energy efficiency. However, similar to many embedded systems, the off-chip communication between the camera and processor presents a bottleneck in terms of power consumption. Inspired by the predictive coding model and expectation suppression phenomenon found in human brain, we propose a temporal attention mechanism to throttle the camera output and pay attention to it only when the visual events cannot be well predicted. The predictive attention not only reduces power consumption in the sensor-processor interface but also effectively decreases the computational workload by filtering out noisy events. We demonstrate that the predictive attention can reduce 46.7% of data communication between the camera and the processor and reduce 43.8% computation activities in the processor.
Paper Structure (14 sections, 9 equations, 9 figures, 2 tables)

This paper contains 14 sections, 9 equations, 9 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND AND RELATED WORKS
  3. DVS data representation
  4. Prediction Models
  5. Attention mechanism
  6. METHOD
  7. Visual Event Predictor
  8. Measuring Event Similarity
  9. Prediction Evaluator and attention generator
  10. Experiment Result
  11. Experimental Setup
  12. Predictor Performance
  13. Attention Directed Situation Awareness
  14. Conclusion

Figures (9)

  • Figure 1: Overall visual predictive attention system architecture.
  • Figure 2: Visual event predictor.
  • Figure 3: The prediction can reduce the noise and reflect a region activity
  • Figure 4: The prediction can reduce the noise and reflect a region activity. The left most figure is the reference, others are figures where the ball was moved to the right by a percentage of radius(the black box denotes the reference position).
  • Figure 5: Esim estimation model
  • ...and 4 more figures