NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes

TL;DR

NeSF introduces a two-stage pipeline that learns 3D semantic fields from posed RGB images by first fitting per-scene NeRF density fields and then translating density into 3D semantic distributions via a learned, shared network. The semantic field can be rendered into 2D semantic maps from novel viewpoints and queried directly in 3D, all trained with only 2D semantic supervision. The approach generalizes to unseen scenes, demonstrated on three synthetic Kubric-based datasets, and is complemented by ablations and qualitative analyses highlighting its strengths in multi-view consistency and geometry-driven labeling, as well as its limitations with thin structures. This work advances scalable 3D semantic understanding by removing the need for 3D supervision at training or test time and provides public datasets to spur further research.

Abstract

We present NeSF, a method for producing 3D semantic fields from posed RGB images alone. In place of classical 3D representations, our method builds on recent work in implicit neural scene representations wherein 3D structure is captured by point-wise functions. We leverage this methodology to recover 3D density fields upon which we then train a 3D semantic segmentation model supervised by posed 2D semantic maps. Despite being trained on 2D signals alone, our method is able to generate 3D-consistent semantic maps from novel camera poses and can be queried at arbitrary 3D points. Notably, NeSF is compatible with any method producing a density field, and its accuracy improves as the quality of the density field improves. Our empirical analysis demonstrates comparable quality to competitive 2D and 3D semantic segmentation baselines on complex, realistically rendered synthetic scenes. Our method is the first to offer truly dense 3D scene segmentations requiring only 2D supervision for training, and does not require any semantic input for inference on novel scenes. We encourage the readers to visit the project website.

Figures (14)

• Figure 1: Overview -- We train our method on collections of posed 2D RGB images and 2D semantic maps, each collection describing an independent scene. Given a new set of posed 2D RGB images, we extract an implicit volumetric representation of the scene's 3D geometry and infer a 3D semantic field. The semantic field can then be used to render dense 2D semantic maps from novel camera poses or queried directly in 3D. Our method generalizes to novel scenes, and requires as little as one semantic map per scene at training time. We encourage readers to visit the \ProjectWebsiteLink.
• Figure 2: Architecture -- Given a pre-trained NeRF model, we sample its volumetric density grid to obtain the 3D scene representation. This grid is converted to a semantic-feature grid by employing a fully convolutional volume-to-volume network thus allowing for geometric reasoning. The semantic-feature grid is in turn translated to semantic probability distributions using the volumetric rendering equation. Note the semantic 3D UNet is trained across all scenes in the train scenes set, though not explicitly depicted for the sake of simplicity. Additionally, note that NeSF is trained solely using 2D supervisory signals and that no segmentation maps are provided at test time.
• Figure 3: Dataset examples -- Each frame includes an RGB image, semantic map, and depth map (not pictured here).
• Figure 4: Ablation: data efficiency -- 2D and 3D mIoU as a function of the number of train scenes for scenes with supervision from 1, 2, 5, 10, or 25 semantic maps per scene. NeSF generalizes to new scenes with as few as a one semantic map per scene. Additional semantic maps per scene marginally improve the accuracy. Experiments on KLEVR dataset.
• Figure 5: Qualitative comparison (ToyBox13) -- Unlike DeepLab, NeSF is able to clearly separate objects with similar appearance but different geometry (Top). However, NeSF struggles with thin structures like lamp posts (Middle) and tends to smear labels from nearby objects (Bottom). SparseConvNet suffers from neither limitation but has access to oracle 3D geometry and full 3D supervision.