Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BlinkFlow: A Dataset to Push the Limits of Event-based Optical Flow Estimation

Yijin Li, Zhaoyang Huang, Shuo Chen, Xiaoyu Shi, Hongsheng Li, Hujun Bao, Zhaopeng Cui, Guofeng Zhang

TL;DR

A novel simulator, BlinkSim, for the fast generation of large-scale data for event-based optical flow, which improves the generalization performance of state-of-the-art methods by more than 40% on average and up to 90%.

Abstract

Event cameras provide high temporal precision, low data rates, and high dynamic range visual perception, which are well-suited for optical flow estimation. While data-driven optical flow estimation has obtained great success in RGB cameras, its generalization performance is seriously hindered in event cameras mainly due to the limited and biased training data. In this paper, we present a novel simulator, BlinkSim, for the fast generation of large-scale data for event-based optical flow. BlinkSim incorporates a configurable rendering engine alongside an event simulation suite. By leveraging the wealth of current 3D assets, the rendering engine enables us to automatically build up thousands of scenes with different objects, textures, and motion patterns and render very high-frequency images for realistic event data simulation. Based on BlinkSim, we construct a large training dataset and evaluation benchmark BlinkFlow that contains sufficient, diversiform, and challenging event data with optical flow ground truth. Experiments show that BlinkFlow improves the generalization performance of state-of-the-art methods by more than 40\% on average and up to 90\%. Moreover, we further propose an Event-based optical Flow transFormer (E-FlowFormer) architecture. Powered by our BlinkFlow, E-FlowFormer outperforms the SOTA methods by up to 91\% on the MVSEC dataset and 14\% on the DSEC dataset and presents the best generalization performance. The source code and data are available at https://zju3dv.github.io/blinkflow/.

BlinkFlow: A Dataset to Push the Limits of Event-based Optical Flow Estimation

TL;DR

A novel simulator, BlinkSim, for the fast generation of large-scale data for event-based optical flow, which improves the generalization performance of state-of-the-art methods by more than 40% on average and up to 90%.

Abstract

Event cameras provide high temporal precision, low data rates, and high dynamic range visual perception, which are well-suited for optical flow estimation. While data-driven optical flow estimation has obtained great success in RGB cameras, its generalization performance is seriously hindered in event cameras mainly due to the limited and biased training data. In this paper, we present a novel simulator, BlinkSim, for the fast generation of large-scale data for event-based optical flow. BlinkSim incorporates a configurable rendering engine alongside an event simulation suite. By leveraging the wealth of current 3D assets, the rendering engine enables us to automatically build up thousands of scenes with different objects, textures, and motion patterns and render very high-frequency images for realistic event data simulation. Based on BlinkSim, we construct a large training dataset and evaluation benchmark BlinkFlow that contains sufficient, diversiform, and challenging event data with optical flow ground truth. Experiments show that BlinkFlow improves the generalization performance of state-of-the-art methods by more than 40\% on average and up to 90\%. Moreover, we further propose an Event-based optical Flow transFormer (E-FlowFormer) architecture. Powered by our BlinkFlow, E-FlowFormer outperforms the SOTA methods by up to 91\% on the MVSEC dataset and 14\% on the DSEC dataset and presents the best generalization performance. The source code and data are available at https://zju3dv.github.io/blinkflow/.
Paper Structure (14 sections, 3 equations, 6 figures, 7 tables)

This paper contains 14 sections, 3 equations, 6 figures, 7 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Optical Flow Dataset and Simulator for Event Camera
  4. Event-based Optical Flow Estimation
  5. BlinkSim
  6. Event Generation Model
  7. Rendering Engine
  8. Event Data Simulator
  9. BlinkFlow Dataset
  10. E-FlowFormer
  11. E-RAFT Brief Review
  12. Event Feature Enhancement Through Transformers
  13. EXPERIMENT
  14. CONCLUSION

Figures (6)

  • Figure 1: Comparison of generalization. E-RAFT eraft overfits the DSEC dataset dsec and generalizes poorly to unseen environments. In contrast, our E-FlowFormer powered by the proposed BlinkFlow dataset presents good generalization performance and recovers complex and flexible optical flows.
  • Figure 2: Video interpolation artifacts (pointed by red arrows).
  • Figure 3: Example scenes from DSEC, MVSEC and the proposed BlinkFlow dataset. 3rd row: Event data, 4th row: Optical flow images. Our BlinkFlow dataset contains complex object/camera motions and various scenarios which significantly outperform DSEC and MVSEC in quantity, quality, and diversity. The 1-2, 3-4, and 5-6 columns of the BlinkFlow Dataset correspond to the sequences of FlyingObjects, E-Tartan and E-Blender, respectively. Best viewed on a color screen in high resolution.
  • Figure 4: Framework of BlinkSim. BlickSim consists of a configurable rendering engine and an event data simulation suite. It allows the fast generation of large-scale data with realistic event data and related optical flow ground truth.
  • Figure 5: Network architecture of E-FlowFormer. The proposed feature enhancement module brings a better correlation volume through the transformer design. Besides, it provides a better initialization of optical flow for the following refinement module.
  • ...and 1 more figures