eCARLA-scenes: A synthetically generated dataset for event-based optical flow prediction
Jad Mansour, Hayat Rajani, Rafael Garcia, Nuno Gracias
TL;DR
This work tackles the shortage of diverse, labeled event-based datasets for optical flow in autonomous driving. It introduces eCARLA-scenes, a CARLA-based synthetic dataset of event streams, grayscale frames, and ground-truth optical flow, alongside the eWiz library for end-to-end event data processing. It details data generation pipelines, sensor configurations, and data processing modules for encoding, augmentation, visualization, and evaluation, all designed for seamless integration with PyTorch and Tonic. The result is a scalable, reproducible platform that enables advanced event-based vision research and paves the way for deployment on neuromorphic hardware.
Abstract
The joint use of event-based vision and Spiking Neural Networks (SNNs) is expected to have a large impact in robotics in the near future, in tasks such as, visual odometry and obstacle avoidance. While researchers have used real-world event datasets for optical flow prediction (mostly captured with Unmanned Aerial Vehicles (UAVs)), these datasets are limited in diversity, scalability, and are challenging to collect. Thus, synthetic datasets offer a scalable alternative by bridging the gap between reality and simulation. In this work, we address the lack of datasets by introducing eWiz, a comprehensive library for processing event-based data. It includes tools for data loading, augmentation, visualization, encoding, and generation of training data, along with loss functions and performance metrics. We further present a synthetic event-based datasets and data generation pipelines for optical flow prediction tasks. Built on top of eWiz, eCARLA-scenes makes use of the CARLA simulator to simulate self-driving car scenarios. The ultimate goal of this dataset is the depiction of diverse environments while laying a foundation for advancing event-based camera applications in autonomous field vehicle navigation, paving the way for using SNNs on neuromorphic hardware such as the Intel Loihi.