NeuroPump: Simultaneous Geometric and Color Rectification for Underwater Images

TL;DR

NeuroPump tackles the problem of underwater image restoration by jointly rectifying geometric distortions caused by refraction and color distortions from absorption and scattering within a self-supervised NeRF-based framework. By explicitly modeling refraction with Snell's law and decoupling water properties (attenuation $oldsymbol{eta}$ and global background light $oldsymbol{A}$) from scene radiance, it enables simultaneous geometry and color rectification as well as novel-view synthesis under new optical parameters. The method leverages a mip-NeRF 360 backbone and introduces a real paired 360-degree underwater dataset to enable accurate evaluation, demonstrating state-of-the-art performance over baselines on both geometry and color restoration. This work facilitates more reliable underwater imaging applications and supports controllable synthesis of underwater views with adjustable optical conditions, advancing practical depth-aware restoration and virtual viewing capabilities.

Abstract

Underwater image restoration aims to remove geometric and color distortions due to water refraction, absorption and scattering. Previous studies focus on restoring either color or the geometry, but to our best knowledge, not both. However, in practice it may be cumbersome to address the two rectifications one-by-one. In this paper, we propose NeuroPump, a self-supervised method to simultaneously optimize and rectify underwater geometry and color as if water were pumped out. The key idea is to explicitly model refraction, absorption and scattering in Neural Radiance Field (NeRF) pipeline, such that it not only performs simultaneous geometric and color rectification, but also enables to synthesize novel views and optical effects by controlling the decoupled parameters. In addition, to address issue of lack of real paired ground truth images, we propose an underwater 360 benchmark dataset that has real paired (i.e., with and without water) images. Our method clearly outperforms other baselines both quantitatively and qualitatively. Our project page is available at: https://ygswu.github.io/NeuroPump.github.io/.

Figures (7)

• Figure 1: NeuroPump pipeline. NeuroPump begins by applying the Snell's law to mitigate refraction in ray sampling. First, PropMLP takes coarse samples to get approximate object weight $\tilde{w}_{\textit{o}}$. Next, $\tilde{w}_{\textit{o}}$ is used to take NerfMLP resamples to obtain fine object color $c_{\textit{o}}$ and weight $w_{\textit{o}}$. Two scattering parameters, namely the global background light $\mathbf{A}$ and the water attenuation coefficients $\boldsymbol{\beta}$, are trained together with MLPs. $\mathbf{A_0}$ and $\boldsymbol{\beta_0}$ represent the initial values. Third, gamma correction is applied after volume rendering, to convert from linear RGB space to sRGB space. All parameters are optimized mainly by minimizing the loss between the model inferred and camera-captured multi-view underwater images.
• Figure 2: Underwater imaging process. Right box is the zoom-in of Left box. Let $o$ be the camera optical center, $s$ and $z-s$ denote the perpendicular distances between the camera optical center and object point to the lens case interface, respectively. An underwater light ray $\mathbf{r}_{\text{w}}$ with direction $\mathbf{d}_\text{w}$ first refracts at the camera lens case interface, and travels along direction $\mathbf{d}_\text{a}$, and finally passes through the camera optical center and hits the sensor at pixel $\boldsymbol{x}$. The angle of incidence and the angle of refraction are $\phi_{\text{w}}$ and $\phi_{\text{a}}$, respectively. Note that the directions of $\mathbf{d}_\text{w}$ and $\mathbf{d}_{\text{a}}$ are reversed in NeuroPump pipeline in \ref{['sec:method']}.
• Figure 3: Rectified underwater images and intermediate results. The 1st row is the underwater image with geometric rectification only. Clearly, the original SeaThru-NeRF's result looks like a zoomed-in version of the in-air ground truth, because it cannot rectify refraction. The 2nd row is the underwater image with joint geometric and color rectification (the final target). Note that SeaThru-NeRF and SeaThru-NeRF (Lav) cannot accurately rectify underwater color, and the estimated depth (the 3rd row) and back-scatter (the 4th row) are also inferior. Mip360 (Lav)-based methods show differences in brightness and saturation compared to the in-air ground truth. In comparison, our NeuroPump shows clear advantages in all results. See more setups and results in Supp..
• Figure 4: Novel view and optical parameter synthesis. The camera pose and the perpendicular distance from optical center to the lens case interface $s$, the medium refractive index $n_{\text{new}}$, and global background light $\textbf{A}$ are varied for new image synthesis.
• Figure 5: Underwater imaging. We assume the world origin is at the camera optical center $O=(0,0,0)$, and the camera sensor's principle point is $m=(0,0)$. Giving an object point $\mathbf{x}=(x, y, z)$ in the world space, its corresponding pixel coordinate is $\boldsymbol{x}=(u,v)$ when imaged underwater due to refraction. While $\mathbf{x}$'s actual pixel coordinate should be $\boldsymbol{x}'=(u',v')$. Moreover, $\mathbf{g}$ is the normal vector of lens case interface.