PruNeRF: Segment-Centric Dataset Pruning via 3D Spatial Consistency

TL;DR

Distractors in NeRF training images can corrupt 3D supervision and degrade view synthesis quality. PruNeRF introduces a segment-centric pruning framework that combines pixel-wise distraction scores with 3D spatial consistency via depth-based reprojection, and further refines distractor localization using zero-shot segmentation (e.g., SAM) at the segment level. The method leverages Influence Functions to measure pixel-wise distraction, 3D consistency checks to identify 3D-aware distractors, and pixel-to-segment refinement to prune distractors accurately, achieving state-of-the-art robustness on synthetic and natural datasets. This approach reduces dataset construction burdens for NeRF and enhances practical robustness in real-world scenes.

Abstract

Neural Radiance Fields (NeRF) have shown remarkable performance in learning 3D scenes. However, NeRF exhibits vulnerability when confronted with distractors in the training images -- unexpected objects are present only within specific views, such as moving entities like pedestrians or birds. Excluding distractors during dataset construction is a straightforward solution, but without prior knowledge of their types and quantities, it becomes prohibitively expensive. In this paper, we propose PruNeRF, a segment-centric dataset pruning framework via 3D spatial consistency, that effectively identifies and prunes the distractors. We first examine existing metrics for measuring pixel-wise distraction and introduce Influence Functions for more accurate measurements. Then, we assess 3D spatial consistency using a depth-based reprojection technique to obtain 3D-aware distraction. Furthermore, we incorporate segmentation for pixel-to-segment refinement, enabling more precise identification. Our experiments on benchmark datasets demonstrate that PruNeRF consistently outperforms state-of-the-art methods in robustness against distractors.

Figures (9)

• Figure 1: Illustration of PruNeRF, which includes measuring pixel-wise distraction scores and then assessing 3D spatial consistency using a depth-based reprojection technique. The red star and green stars denote a query pixel and projected pixels, respectively.
• Figure 2: Comparison of pixel-wise measurements in two natural scenes: Statue and Baby Yoda. In (a), the boundaries of distractors are highlighted in red. In (b) and (c), the heatmaps of loss and gradient-norm are relatively noisy and struggle to discern distractors from hard-to-learn regions. However, in (d), Influence Function reveals a more precise detection of distractors compared with others.
• Figure 3: Visualization of each step in PruNeRF. (a) Training image with distractors highlighted in red. (b) Influence heatmaps identifying distractor pixels (\ref{['subsec:if']}). (c) 3D spatial consistency further refines distractor identification (\ref{['subsec:consistency']}). (d) and (e) show the segmentation results and our final mask of distractors through pixel-to-segment refinement, respectively (\ref{['subsec:seg']}).
• Figure 4: Qualitative evaluation on Statue, Android, and BabyYoda. PruNeRF exhibits superior performance, especially in hard-to-learn regions highlighted in the yellow box. Additional results are provided in \ref{['appen:qual_eval']}.
• Figure 5: Qualitative evaluation on Pick. This dataset includes a moving hand as a distractor that changes its location with temporal continuity. PruNeRF demonstrates enhanced performance, particularly in hard-to-learn regions as emphasized in the box, compared with the state-of-the-art methods.