RecGS: Removing Water Caustic with Recurrent Gaussian Splatting

TL;DR

This work tackles water caustics in shallow-water seafloor imaging by introducing Recurrent Gaussian Splatting (RecGS), a framework that builds a dense 3D representation via 3D Gaussian Splatting (3DGS) and progressively removes caustics through an iterative, low-pass residual decomposition. Caustics are modeled as additive radiance $C$ and estimated from low-frequency components of the image residual $R = I - \hat{I}$ using a 2D FFT, with $C = \text{ifft}(\text{fft}(R)_{[0:k]})$ and $k = 9$, and the 3DGS model is refined recurrently to minimize $\text{dist}(I - C, \hat{I})$. Compared to joint optimization, 2D filtering, and pretrained deep-learning approaches, RecGS yields more balanced illumination and photorealistic underwater renderings without annotated data, demonstrating robustness to illumination inconsistencies in real scenes. The method holds promise for broader 3D vision tasks with challenging lighting and sparse data, particularly where 3D structure can guide illumination correction.

Abstract

Water caustics are commonly observed in seafloor imaging data from shallow-water areas. Traditional methods that remove caustic patterns from images often rely on 2D filtering or pre-training on an annotated dataset, hindering the performance when generalizing to real-world seafloor data with 3D structures. In this paper, we present a novel method Recurrent Gaussian Splatting (RecGS), which takes advantage of today's photorealistic 3D reconstruction technology, 3DGS, to separate caustics from seafloor imagery. With a sequence of images taken by an underwater robot, we build 3DGS recurrently and decompose the caustic with low-pass filtering in each iteration. In the experiments, we analyze and compare with different methods, including joint optimization, 2D filtering, and deep learning approaches. The results show that our method can effectively separate the caustic from the seafloor, improving the visual appearance, and can be potentially applied on more problems with inconsistent illumination.

Figures (9)

• Figure 1: Underwater imaging in shallow water suffers from water caustics. Our proposed pipeline based on 3DGS and Fourier low-rank reconstruction decomposes the background from water caustic without supervision.
• Figure 2: Related Works: Classic methods (Left) are based on image filtering along a certain time window on the 2D image space. The performance gets downgraded when a 3D structure is present in the scene, or if the camera moves fast and observes less overlap between frames. Deep learning methods (Mid) require expert annotations, which is extremely expensive to scale up. Neural networks trained with limited annotated data do not generalize well to novel observations. Our proposed method (Right) maintains dense 3D scene representations by building a 3D Gaussian model recurrently and decomposing low-frequency caustics from residuals. Our method works well on images captured from an underwater robot in the wild, without any pretraining on a dataset.
• Figure 3: Our proposed recurrent 3DGS workflow: we build a vanilla 3DGS model with caustics in the images first, then find the residual between captured image and rendered view. We run 2D FFT on the residual image, and reconstruct it with only the low-rank part. This low-rank reconstruction is then subtracted from the training images.
• Figure 4: We collected data in real world marine environment in Florida Keys area. The Robot we use is a LSU's Bruce ROV bruce2023 equipped with ZED cameras.
• Figure 5: Jointly optimizing a low-rank Fourier spectrum together with 3DGS leads to an ill-posed behavior. As shown in dashed box, joint-optimization method creates undesired over-exposed areas, while still maintaining multi-view consistency. In comparison, our recurrent method restores the scene with uniform illumination.