ODYSSEE: Oyster Detection Yielded by Sensor Systems on Edge Electronics

TL;DR

This work tackles the challenge of monitoring oyster reefs with minimal disturbance and scalable data collection by combining synthetic data generation with real underwater imagery. A diffusion-based sim-to-real pipeline, conditioned by ControlNet and Blender-derived geometry, augments a real oyster dataset to train a YOLOv10 detector that runs on the Aqua2 AUV. The approach yields a state-of-the-art mAP@50 of approximately 0.657 on edge hardware, demonstrating viability for real-time, onboard oyster surveys and habitat assessment. The methods enable scalable, non-destructive monitoring and provide a foundation for broader autonomous marine environmental sensing.

Abstract

Oysters are a vital keystone species in coastal ecosystems, providing significant economic, environmental, and cultural benefits. As the importance of oysters grows, so does the relevance of autonomous systems for their detection and monitoring. However, current monitoring strategies often rely on destructive methods. While manual identification of oysters from video footage is non-destructive, it is time-consuming, requires expert input, and is further complicated by the challenges of the underwater environment. To address these challenges, we propose a novel pipeline using stable diffusion to augment a collected real dataset with realistic synthetic data. This method enhances the dataset used to train a YOLOv10-based vision model. The model is then deployed and tested on an edge platform in underwater robotics, achieving a state-of-the-art 0.657 mAP@50 for oyster detection on the Aqua2 platform.

Figures (6)

• Figure 1: Oyster detection system deployed on an Autonomous Underwater Vehicle (AUV). The main image shows oyster detection in real-time in a shallow marine environment on the Aqua2 robot. The inset highlights the output bounding boxes around the oysters detected in situ
• Figure 2: Overview of the oyster detection system. Simulated oyster assemblages are rendered using Blender and paired with ground-truth segmentation masks. Diffusion models are employed to generate synthetic oyster imagery, ensuring consistency with real oyster habitats. Real-world data are collected by the Aqua2 and BlueROV, equipped with a camera for capturing oyster images in the field. The combined dataset is used to train a YOLOv10, which is deployed on an edge computing platform on Aqua2 for real-time oyster detection.
• Figure 3: Experimental deployment of the system, surveying a wild near-shore oyster reef in Lewes, DE. Operators can be seen teleoperating the Aqua2 and analyzing the feedback.
• Figure 4: Synthetic image generation pipeline using Stable Diffusion and ControlNet. ControlNet uses randomly sampled real underwater images combined with Blender-geneated images, depth images, and ground truth masks to ensure consistency. Stable Difussion model, guided by text prompts, and ControlNet, refines the synthetic data to closely match real-world oyster environments for vision model training.
• Figure 5: Qualatative Comparison of All YOLOv10 Sub-Model Performance. This image is best seen in color on a computer screen at 400% zoom.