IDDR-NGP: Incorporating Detectors for Distractor Removal with Instant Neural Radiance Field

TL;DR

This work tackles the problem of removing diverse distractors from 3D scenes reconstructed via implicit representations. It introduces IDDR-NGP, which integrates 2D detectors with Instant-NGP, masking samples inside detected bounding boxes and optimizing with a multi-view compensation loss and a perceptual LPIPS loss to ensure cross-view consistency. Key contributions include the first unified distractor removal method for 3D scenes, a new synthetic-real distractor benchmark, and extensive experiments showing robustness across multiple distractor types and comparable performance to state-of-the-art desnow methods. The approach enables efficient end-to-end 3D restoration from multi-view corrupted images, with practical impact for robust 3D perception in robotics and related vision tasks.

Abstract

This paper presents the first unified distractor removal method, named IDDR-NGP, which directly operates on Instant-NPG. The method is able to remove a wide range of distractors in 3D scenes, such as snowflakes, confetti, defoliation and petals, whereas existing methods usually focus on a specific type of distractors. By incorporating implicit 3D representations with 2D detectors, we demonstrate that it is possible to efficiently restore 3D scenes from multiple corrupted images. We design the learned perceptual image patch similarity~( LPIPS) loss and the multi-view compensation loss (MVCL) to jointly optimize the rendering results of IDDR-NGP, which could aggregate information from multi-view corrupted images. All of them can be trained in an end-to-end manner to synthesize high-quality 3D scenes. To support the research on distractors removal in implicit 3D representations, we build a new benchmark dataset that consists of both synthetic and real-world distractors. To validate the effectiveness and robustness of IDDR-NGP, we provide a wide range of distractors with corresponding annotated labels added to both realistic and synthetic scenes. Extensive experimental results demonstrate the effectiveness and robustness of IDDR-NGP in removing multiple types of distractors. In addition, our approach achieves results comparable with the existing SOTA desnow methods and is capable of accurately removing both realistic and synthetic distractors.

Figures (6)

• Figure 1: The framework of our method IDDR-NGP. In the component of hash encoding, different colors denote different resolutions and the corresponding embedding vectors.
• Figure 2: Anchor-free network structure of FCOS.
• Figure 3: Anchor-based network structure of YOLOv5.
• Figure 4: Qualitative results on synthetic scenes. Given the input images, we visualize the desnow results of Weather removal, SnowFormer, HCDW-Net and our IDDR-NGP in a consistent view.
• Figure 5: Qualitative results on realistic scenes with real distractors. We compare our method with IDDR-Inpainting$^1$ and IDDR-Inpainting$^2$. Note that CR-Fill is utilized as the inpainting backbone in IDDR-Inpainting$^1$ and T-Fill is adopted as the inpainting backbone in IDDR-Inpainting$^2$.