Relative Illumination Fields: Learning Medium and Light Independent Underwater Scenes

TL;DR

This work tackles underwater NeRFs under inhomogeneous, co-moving illumination and scattering by introducing a camera-attached local illumination field that co-moves with the vehicle and a volumetric medium rendering component. The method disentangles light distribution, object color, and water medium effects by learning an illumination factor $\alpha$ from local position and surface normal, while separately modeling attenuation and backscatter within a unified rendering equation. The approach is calibration-free and supports arbitrary light configurations, demonstrated on synthetic and real datasets with multiple lights, showing effective color restoration and high-fidelity view synthesis in challenging underwater scenes. This has practical impact for underwater robotics and imaging, enabling robust 3D scene understanding in turbid, dark or dynamic lighting without explicit light sensing or calibration, albeit with limitations on shadow modeling and pose diversity.

Abstract

We address the challenge of constructing a consistent and photorealistic Neural Radiance Field in inhomogeneously illuminated, scattering environments with unknown, co-moving light sources. While most existing works on underwater scene representation focus on a static homogeneous illumination, limited attention has been paid to scenarios such as when a robot explores water deeper than a few tens of meters, where sunlight becomes insufficient. To address this, we propose a novel illumination field locally attached to the camera, enabling the capture of uneven lighting effects within the viewing frustum. We combine this with a volumetric medium representation to an overall method that effectively handles interaction between dynamic illumination field and static scattering medium. Evaluation results demonstrate the effectiveness and flexibility of our approach.

Figures (6)

• Figure 1: Our proposed approach reconstructs a neural radiance field of an underwater scene captured by a camera system with unknown, co-moving light sources, enabling novel view synthesis (a). Additionally, it can recover the clean scene representation where neither medium nor light cone effects are present (b) and is capable of disentangling the light cone in the scattering medium (c) and the surface illumination (d).
• Figure 2: Autonomous underwater vehicle in $100m$ depth in coastal waters. Note the illumination pattern at the seafloor as well as the scattering (the light cone) in the water.
• Figure 3: An illustration of the problem setup and the architecture of our proposed approach. The global NeRF MLP $\mathcal{F}_{\Theta}$ learns both density and color at each ray sample in the world coordinate frame. Simultaneously, each ray sample is transformed into the local camera coordinate frame, with surface normals derived from the predicted density field, allowing the local illumination field MLP $\mathcal{F}^l_{\Theta}$ to estimate the light intensity factor $\hbox{\boldmath$\alpha$}$ that the sample point receives. Finally, medium-related parameters, that is the attenuation coefficient $\hbox{\boldmath$\sigma$}_{\mathrm{attn}}$, medium color $\hbox{\boldmath$c$}_{\mathrm{med}}$, and backscatter $\hbox{\boldmath$\sigma$}_{\mathrm{bs}}$ are jointly estimated and integrated together into a underwater volume rendering formulation.
• Figure 4: Experiments on different datasets. Top two rows are in air, bottom three in water. Sets contain one, four, two, two and one co-moving light from top to bottom. The Four Lights and Color Checker dataset are rendered, the Tank dataset is captured by us, the DarkGS zhang2024darkgs and Beyond Nerf: Underwater zhang2023beyond datasets are freely available. Since we are tackling a largely unsolved problem, there are virtually no competitor methods to compare against. We found it still instructive to present results obtained by the methods marked with an asterisk that do not explicitly model co-moving light but are still related to our scenario in some aspects.
• Figure 5: Ablation of single- and three-channel illumination field. Estimating three channels for the illumination field has is advantageous for in medium data. It is mostly irrelevant for in-air data. To represent the received light at a certain scene point the field must be able to model the channel dependent attenuation of the light on its way to the object.