Fast Underwater Scene Reconstruction using Multi-View Stereo and Physical Imaging

TL;DR

This work tackles underwater scene reconstruction by marrying traditional Multi-View Stereo (MVS) depth estimation with a physics-based underwater image formation model. The proposed two-branch framework includes an MVS depth pathway and a medium-parameter rendering pathway, complemented by a color MLP, enabling both high-fidelity geometry and true-color restoration without requiring ground-truth depth data. Empirical results on SeaThru-NeRF datasets show improved rendering quality and significantly faster training compared to NeRF-based underwater methods, with ablations highlighting the pivotal role of the medium subnet. The approach provides a practical, efficient solution for underwater novel-view synthesis and color restoration in scattering media, with potential applications in marine science and underwater analytics.

Abstract

Underwater scene reconstruction poses a substantial challenge because of the intricate interplay between light and the medium, resulting in scattering and absorption effects that make both depth estimation and rendering more complex. While recent Neural Radiance Fields (NeRF) based methods for underwater scenes achieve high-quality results by modeling and separating the scattering medium, they still suffer from slow training and rendering speeds. To address these limitations, we propose a novel method that integrates Multi-View Stereo (MVS) with a physics-based underwater image formation model. Our approach consists of two branches: one for depth estimation using the traditional cost volume pipeline of MVS, and the other for rendering based on the physics-based image formation model. The depth branch improves scene geometry, while the medium branch determines the scattering parameters to achieve precise scene rendering. Unlike traditional MVSNet methods that rely on ground-truth depth, our method does not necessitate the use of depth truth, thus allowing for expedited training and rendering processes. By leveraging the medium subnet to estimate the medium parameters and combining this with a color MLP for rendering, we restore the true colors of underwater scenes and achieve higher-fidelity geometric representations. Experimental results show that our method enables high-quality synthesis of novel views in scattering media, clear views restoration by removing the medium, and outperforms existing methods in rendering quality and training efficiency.

Figures (4)

• Figure 1: Overview. We first use an FPN to extract image features from the source view and warp them into warped features $\{F_{i}^{w}\}_{i=1}^{N}$. The distorted characteristics are combined into a cost volume, which is then processed using a 3D CNN to generate depth. Subsequently, a pooling network is utilized to consolidate features for each 3D point at the predicted depth and then integrate them with the source image features. Following this, a color MLP and a medium-sized subnet are utilized to analyze these features, resulting in a refined image with the medium removed. Ultimately, the medium subnet is utilized once more to conduct additional processing on the image, resulting in the extraction of the backscatter and attenuation images. These are subsequently combined to produce the ultimate reconstructed outcome.
• Figure 2: Restorations and Depth Maps. We contrast our approach with SeaThru-NeRF through the exhibition of renderings devoid of the medium. Underneath each image, we present the respective depth maps. Our restoration technique effectively preserves a greater level of color detail. In comparison to SeaThru-NeRF, the depth renderings exhibit a higher level of smoothness and coherence.
• Figure 3: The rendering of distant details. We compare our method with several baseline methods. Our method outperforms them in rendering quality and better preserves distant geometric details.
• Figure 4: Novel view synthesis in water medium. The columns, from left to right, show the underwater scene renderings for the 'Cueaçao', 'IUI3 Red Sea', 'Japanese Gardens Red Sea', and 'Panama' scenes. The rendering quality of distant details, highlighted within the red squares, is compared and presented in Fig. \ref{['fig:detail']}.