SurfSLAM: Sim-to-Real Underwater Stereo Reconstruction For Real-Time SLAM

TL;DR

A novel framework that enables sim-to-real training of underwater stereo disparity estimation networks using simulated data and self-supervised finetuning is proposed and demonstrates the advantages of the proposed training approach on real-world data for improving stereo estimation in the underwater domain and for enabling accurate trajectory estimation and 3D reconstruction of complex shipwreck sites.

Abstract

Localization and mapping are core perceptual capabilities for underwater robots. Stereo cameras provide a low-cost means of directly estimating metric depth to support these tasks. However, despite recent advances in stereo depth estimation on land, computing depth from image pairs in underwater scenes remains challenging. In underwater environments, images are degraded by light attenuation, visual artifacts, and dynamic lighting conditions. Furthermore, real-world underwater scenes frequently lack rich texture useful for stereo depth estimation and 3D reconstruction. As a result, stereo estimation networks trained on in-air data cannot transfer directly to the underwater domain. In addition, there is a lack of real-world underwater stereo datasets for supervised training of neural networks. Poor underwater depth estimation is compounded in stereo-based Simultaneous Localization and Mapping (SLAM) algorithms, making it a fundamental challenge for underwater robot perception. To address these challenges, we propose a novel framework that enables sim-to-real training of underwater stereo disparity estimation networks using simulated data and self-supervised finetuning. We leverage our learned depth predictions to develop \algname, a novel framework for real-time underwater SLAM that fuses stereo cameras with IMU, barometric, and Doppler Velocity Log (DVL) measurements. Lastly, we collect a challenging real-world dataset of shipwreck surveys using an underwater robot. Our dataset features over 24,000 stereo pairs, along with high-quality, dense photogrammetry models and reference trajectories for evaluation. Through extensive experiments, we demonstrate the advantages of the proposed training approach on real-world data for improving stereo estimation in the underwater domain and for enabling accurate trajectory estimation and 3D reconstruction of complex shipwreck sites.

Figures (9)

• Figure 1: SurfSLAM performs real-time SLAM in challenging underwater environments by fusing proprioceptive data with global registration. Accurate global registration is enabled by an underwater stereo disparity estimation algorithm that is fine-tuned via a novel sim-to-real pipeline. Illustrated is our BlueROV data collection platform navigating a shipwreck. Two stereo frames along the robot's path (drawn in pink) are visualized. Background image courtesy of the National Oceanic and Atmospheric Administration Thunder Bay National Marine Sanctuary.
• Figure 2: An overview of SurfSLAM. We take as input measurements from a barometer, IMU, DVL, and a stereo image pair. We maintain an acoustic-inertial pose graph that preintegrates measurements to maintain a pose throughout operation. In parallel, we use our finetuned underwater stereo network to produce metric depth maps. These depth estimates and stereo images are used to perform geometric tracking to perform global registration, and reduce drift over operation, producing accurate trajectories and dense maps during operation.
• Figure 3: Coordinate frames of the robot. Red indicates the $x$-axis, green the $y$-axis, and blue the $z$-axis. The IMU ($\mathtt{I}$) and base ($\mathtt{{B}}$) frames are coincident. The DVL ($\mathtt{D}$) and barometer ($\mathtt{P}$) frames share the same frame convention and the camera frame ($\mathtt{C}$) follows the OpenCV frame convention noauthor_opencv_nodate.
• Figure 4: A demonstration of our underwater augmentation pipeline. From left to right, the first five columns demonstrate increasing levels of augmentation. The rightmost column shows the reference image that the augmentations aim to match.
• Figure 5: An example from three of the nine stereo sequences is shown. The first column shows the left rectified image; the second column shows the disparity estimated from our metric photogrammetry pipeline; the last column shows the manually-annotated foreground mask overlaid on the left image. The disparity color maps are so that lighter (yellow) is higher disparity and darker (purple) is lower disparity, with black being zero disparity.