eStonefish-scenes: A synthetically generated dataset for underwater event-based optical flow prediction tasks

TL;DR

This work tackles the lack of underwater event-based optical flow datasets by introducing eStonefish-scenes, a synthetic dataset generated in the Stonefish simulator for autonomous underwater vehicles. It combines a modular data-generation pipeline (Stonefish-scenegen for coral-rich scenes and Stonefish-boids for fish schooling) with the eWiz processing library to provide end-to-end data creation, encoding, augmentation, visualization, and evaluation. The dataset includes synchronized event streams, grayscale frames, and ground-truth optical flow, across static reef/rocky seabeds and dynamic schooling scenarios, enabling robust training of CNNs and SNNs for odometry, obstacle avoidance, and navigation. By offering scalable scene generation and a unified processing framework, the work facilitates rapid development and benchmarking of underwater event-based motion estimation methods with practical impact for AUVs and related robotics applications.

Abstract

The combined use of event-based vision and Spiking Neural Networks (SNNs) is expected to significantly impact robotics, particularly in tasks like visual odometry and obstacle avoidance. While existing real-world event-based datasets for optical flow prediction, typically captured with Unmanned Aerial Vehicles (UAVs), offer valuable insights, they are limited in diversity, scalability, and are challenging to collect. Moreover, there is a notable lack of labelled datasets for underwater applications, which hinders the integration of event-based vision with Autonomous Underwater Vehicles (AUVs). To address this, synthetic datasets could provide a scalable solution while bridging the gap between simulation and reality. In this work, we introduce eStonefish-scenes, a synthetic event-based optical flow dataset based on the Stonefish simulator. Along with the dataset, we present a data generation pipeline that enables the creation of customizable underwater environments. This pipeline allows for simulating dynamic scenarios, such as biologically inspired schools of fish exhibiting realistic motion patterns, including obstacle avoidance and reactive navigation around corals. Additionally, we introduce a scene generator that can build realistic reef seabeds by randomly distributing coral across the terrain. To streamline data accessibility, we present eWiz, a comprehensive library designed for processing event-based data, offering tools for data loading, augmentation, visualization, encoding, and training data generation, along with loss functions and performance metrics.

Figures (6)

• Figure 1: Generated environment, the environment includes a seabed with feature-rich corals
• Figure 2: Simulated BlueROV2 navigating an underwater environment.
• Figure 3: Visualizations of event-based, grayscale, and flow images for different generated sequences. Event-based images are accumulated over 100ms intervals.
• Figure 4: Discretized underwater environment as an Octree structure.
• Figure 5: Currently implemented modules for the eWiz library.