U2NeRF: Unsupervised Underwater Image Restoration and Neural Radiance Fields

TL;DR

This work presents Unsupervised Underwater Neural Radiance Field U2NeRF, a transformer-based architecture that learns to render and restore novel views conditioned on multi-view geometry simultaneously, and releases an Underwater View Synthesis UVS dataset, containing both synthetically-generated and real-world data.

Abstract

Underwater images suffer from colour shifts, low contrast, and haziness due to light absorption, refraction, scattering and restoring these images has warranted much attention. In this work, we present Unsupervised Underwater Neural Radiance Field U2NeRF, a transformer-based architecture that learns to render and restore novel views conditioned on multi-view geometry simultaneously. Due to the absence of supervision, we attempt to implicitly bake restoring capabilities onto the NeRF pipeline and disentangle the predicted color into several components - scene radiance, direct transmission map, backscatter transmission map, and global background light, and when combined reconstruct the underwater image in a self-supervised manner. In addition, we release an Underwater View Synthesis UVS dataset consisting of 12 underwater scenes, containing both synthetically-generated and real-world data. Our experiments demonstrate that when optimized on a single scene, U2NeRF outperforms several baselines by as much LPIPS 11%, UIQM 5%, UCIQE 4% (on average) and showcases improved rendering and restoration capabilities. Code will be made available upon acceptance.

Figures (8)

• Figure 1: Unlike standalone methods like Radiance Fields (for rendering) and Image Enhancement (for restoration), our method U2NeRF simultaneously restores and renders an underwater scene.
• Figure 2: Overview of U2NeRF: 1) Identify source views for a given target view, 2) Extract features for epipolar points using a trainable U-Net like model, 3) For each ray in the target view, sample points and directly predict a target patch disentangled into scene radiance, direct and backscatter transmission maps, and global background light. 4) The individual components are combined based on the image formation model to reconstruct the underwater image which is used as a self-supervision loss.
• Figure 3: Qualitative results for single-scene rendering. In the Debris scene (row-1), U2NeRF is able to successfully recover and restore fishes, and enhance its visibility. In the Starfish scene (row-2), U2NeRF reconstructs edges with greater detail and even comparable to the non-rendering baselines. In scene2 from 'hard' split (row-3), U2NeRF renders complex, moving structures (rope) with higher visual quality.
• Figure 4: Qualitative results for cross-scene rendering. We visualize the underwater scene (row-1), novel views rendered using the pretrained network (row-2), novel views rendered using the finetuned network across different scenes (from left to right). U2NeRF successfully generalizes across scenes and when finetuned captures more intricate details.
• Figure 5: Illustrative examples of scenes from the UVS Dataset, one scene from each split is shown here.