NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentation

TL;DR

This work proposes a novel paradigm, namely “Heuristics-Guided Segmentation” (HuGS), which signifi-cantly enhances the separation of static scenes from tran-sient distractors by harmoniously combining the strengths of hand-crafted heuristics and state-of-the-art segmentation models, thus significantly transcending the limitations of previous solutions.

Abstract

Neural Radiance Field (NeRF) has been widely recognized for its excellence in novel view synthesis and 3D scene reconstruction. However, their effectiveness is inherently tied to the assumption of static scenes, rendering them susceptible to undesirable artifacts when confronted with transient distractors such as moving objects or shadows. In this work, we propose a novel paradigm, namely "Heuristics-Guided Segmentation" (HuGS), which significantly enhances the separation of static scenes from transient distractors by harmoniously combining the strengths of hand-crafted heuristics and state-of-the-art segmentation models, thus significantly transcending the limitations of previous solutions. Furthermore, we delve into the meticulous design of heuristics, introducing a seamless fusion of Structure-from-Motion (SfM)-based heuristics and color residual heuristics, catering to a diverse range of texture profiles. Extensive experiments demonstrate the superiority and robustness of our method in mitigating transient distractors for NeRFs trained in non-static scenes. Project page: https://cnhaox.github.io/NeRF-HuGS/.

Figures (25)

• Figure 1: Comparison between previous methods and the proposed Heuristics-Guided Segmentation (HuGS) paradigm. When training NeRF with static scenes disturbed by transient distractors, (a) segmentation-based methods rely on prior knowledge and cannot identify unexpected transient objects (e.g., pizza); (b) heuristics-based methods are more generalizable but inaccurate (e.g., tablecloth textures); (c) our method combines their strengths and produces highly accurate transient vs. static separations, thereby significantly improving NeRF results.
• Figure 2: Pipeline of HuGS. (a) Given unordered images of a static scene disturbed by transient distractors as input, we first obtain two types of heuristics. (b) SfM-based heuristics use SfM to distinguish between static (green) and transient features (red). The static features are then employed as point prompts to generate dense masks using SAM. (c) Residual-based heuristics are based on a partially trained NeRF (i.e., trained for several thousands of iterations) that can provide reasonable color residuals. (d) Their combination finally guides SAM again to generate (e) the static map for each input image.
• Figure 3: Performance of HuGS using existing heuristics. (a) is an example training image with a moving red car, and (d) is its segmentation result using SAM. (b, e) are heuristic maps obtained from different partially trained models. (c, f) are static maps produced by our method, where inaccurate heuristics may lead to incorrect results (NeRF-W).
• Figure 4: Performance of RobustNeRF vs. transient objects of different sizes. Transient distractors in the training images are framed in white. A lower quantile (threshold) causes the model to miss small-sized static objects, while a higher quantile prevents the removal of large-sized transient objects.
• Figure 5: Heuristics combination. (b) The SfM-based heuristics $\mathcal{H}_i^{SfM}$ alone captures high-frequency static details (e.g., box textures) well but misses smooth ones (e.g., white chairs). This could be complemented by incorporating (e) residual-based heuristics $\mathcal{H}_i^{CR}$ from a (d) Nerfacto with $5k$ training iterations which does the opposite (f). Their combination (c) covers the full spectrum of static scenes and identifies transient objects (e.g., pink balloon).