DerainNeRF: 3D Scene Estimation with Adhesive Waterdrop Removal

TL;DR

The paper tackles 3D scene reconstruction from images captured through glass with adhesive waterdrops by integrating a pre-trained waterdrop detector with Neural Radiance Fields. DerainNeRF masks waterdrop-occluded regions using AttGAN-generated attention maps and trains NeRF on the remaining pixels to implicitly recover the underlying scene, with a mask-enhancement step that averages attention over multiple views when droplets are camera-fixed. Ablation studies show that including the averaged-attention mask improves reconstruction quality, validating the robustness of the approach. Experiments on synthetic and real datasets demonstrate that DerainNeRF yields higher-quality novel-view renderings and outperforms existing image-based waterdrop removal methods, highlighting its potential for practical 3D reconstruction under adverse weather conditions.

Abstract

When capturing images through the glass during rainy or snowy weather conditions, the resulting images often contain waterdrops adhered on the glass surface, and these waterdrops significantly degrade the image quality and performance of many computer vision algorithms. To tackle these limitations, we propose a method to reconstruct the clear 3D scene implicitly from multi-view images degraded by waterdrops. Our method exploits an attention network to predict the location of waterdrops and then train a Neural Radiance Fields to recover the 3D scene implicitly. By leveraging the strong scene representation capabilities of NeRF, our method can render high-quality novel-view images with waterdrops removed. Extensive experimental results on both synthetic and real datasets show that our method is able to generate clear 3D scenes and outperforms existing state-of-the-art (SOTA) image adhesive waterdrop removal methods.

Figures (7)

• Figure 1: Given a set of waterdrop images (left column), our DerainNeRF estimates 3D scenes and reomves the adhesive waterdrops altogether. It synthesizes clear images (right column) with high quality.
• Figure 2: Training procedure of DerainNeRF. A pre-trained deep waterdrop detector detects waterdrops in input images and generate binary masks, then DerainNeRF utilizes the masks to block waterdrop regions in input images during NeRF training.
• Figure 3: An example of (a) waterdrop image, (b) attention map and (c) generated binary mask from attention map
• Figure 4: Qualitative evaluations of our method against SOTA image waterdrop removal methods on the synthetic dataset. Top to bottom shows different scenes including Tanabata, Factory and Church. We render waterdrop-removed images from clear 3D scenes estimated by our method.
• Figure 5: Qualitative comparisons between different methods with real indoor dataset. The experimental results demonstrate that our method can effectively remove droplets whether the glass with waterdrops is fixed to the scene (top row) or fixed to the camera (bottom row).