Flow-Based Visual Stream Compression for Event Cameras

TL;DR

This work tackles the challenge of high data rates from neuromorphic, event-based vision sensors by proposing Flow-Based Compression (FBC), which uses real-time optical-flow estimates to predict future events and reduce transmission needs in an asynchronous, stream-to-stream framework. The method achieves an average compression ratio of $CR\approx 2.81$ with about $68\%$ of events removed and a median temporal error of $0.48\ \mathrm{ms}$, while maintaining a low spatiotemporal distance of $3.07$ when reconstructions are compared to originals. Furthermore, cascading FBC with LZMA yields substantial gains (up to $CR\approx 29.16$ on some datasets), approaching state-of-the-art compression for non-real-time applications. Real-time prediction is demonstrated on both desktop and embedded platforms, supporting practical deployment in bandwidth- and power-constrained environments, with room for further improvements as optical-flow estimates improve.

Abstract

As the use of neuromorphic, event-based vision sensors expands, the need for compression of their output streams has increased. While their operational principle ensures event streams are spatially sparse, the high temporal resolution of the sensors can result in high data rates from the sensor depending on scene dynamics. For systems operating in communication-bandwidth-constrained and power-constrained environments, it is essential to compress these streams before transmitting them to a remote receiver. Therefore, we introduce a flow-based method for the real-time asynchronous compression of event streams as they are generated. This method leverages real-time optical flow estimates to predict future events without needing to transmit them, therefore, drastically reducing the amount of data transmitted. The flow-based compression introduced is evaluated using a variety of methods including spatiotemporal distance between event streams. The introduced method itself is shown to achieve an average compression ratio of 2.81 on a variety of event-camera datasets with the evaluation configuration used. That compression is achieved with a median temporal error of 0.48 ms and an average spatiotemporal event-stream distance of 3.07. When combined with LZMA compression for non-real-time applications, our method can achieve state-of-the-art average compression ratios ranging from 10.45 to 17.24. Additionally, we demonstrate that the proposed prediction algorithm is capable of performing real-time, low-latency event prediction.

Figures (7)

• Figure 1: Conceptual diagram for flow-based compression system.
• Figure 2: Visualization of the event prediction algorithm. Shaded pixel locations represent the initial (green) and predicted (blue) events generated by the algorithm.
• Figure 3: Characterization of FBC performance across various values of $PT$. The metrics evaluated are CR (a), ER (b), distance (c), and temporal error (d). Results use the bar-square dataset for performance characterization.
• Figure 4: Reconstruction performance over time compared to equivalent ER achieved with random removal of events. Distance values are computed in $5~ms$ cubes and displayed as a $250~ms$ moving average of samples to reduce noise.
• Figure 5: Qualitative FBC results for Shapes (top) and Outdoor Day 1 (bottom) scenarios. Events are accumulated for 10 $ms$ (top) and 16 $ms$ (bottom). ON events are in blue and OFF events are in red. Event stream distance at each sample time is annotated as $d$. In column 5 (green box) we see a larger qualitative error corresponding to a higher event distance. This error is the result of an instance in which the assumptions underlying the event prediction algorithm temporarily do not hold. The compression ratio is computed for a sliding 200 $ms$ window for each scenario.