Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decomposing NeRF for Editing via Feature Field Distillation

Sosuke Kobayashi, Eiichi Matsumoto, Vincent Sitzmann

TL;DR

The paper tackles editing NeRF-based scenes by introducing Distilled Feature Fields (DFF), a 3D feature field distilled from 2D vision-language foundation models (e.g., LSeg, DINO) via volume rendering. DFF enables open-set, query-based semantic decomposition of 3D space, allowing selective editing of regions without re-training the radiance field. Empirically, DFF achieves competitive 3D segmentation against supervised baselines and supports multi-view-consistent edits (appearance, deletion, extraction) guided by text or image queries, with potential integration with optimization-based editors like CLIPNeRF. The work highlights a pathway to transfer rich 2D semantic priors into 3D NeRF representations, expanding editable neural graphics of real-world scenes, while noting limitations tied to teacher quality and geometry noise.

Abstract

Emerging neural radiance fields (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations. However, editing a scene represented by a NeRF is challenging, as the underlying connectionist representations such as MLPs or voxel grids are not object-centric or compositional. In particular, it has been difficult to selectively edit specific regions or objects. In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 3D scenes. We propose to distill the knowledge of off-the-shelf, self-supervised 2D image feature extractors such as CLIP-LSeg or DINO into a 3D feature field optimized in parallel to the radiance field. Given a user-specified query of various modalities such as text, an image patch, or a point-and-click selection, 3D feature fields semantically decompose 3D space without the need for re-training and enable us to semantically select and edit regions in the radiance field. Our experiments validate that the distilled feature fields (DFFs) can transfer recent progress in 2D vision and language foundation models to 3D scene representations, enabling convincing 3D segmentation and selective editing of emerging neural graphics representations.

Decomposing NeRF for Editing via Feature Field Distillation

TL;DR

The paper tackles editing NeRF-based scenes by introducing Distilled Feature Fields (DFF), a 3D feature field distilled from 2D vision-language foundation models (e.g., LSeg, DINO) via volume rendering. DFF enables open-set, query-based semantic decomposition of 3D space, allowing selective editing of regions without re-training the radiance field. Empirically, DFF achieves competitive 3D segmentation against supervised baselines and supports multi-view-consistent edits (appearance, deletion, extraction) guided by text or image queries, with potential integration with optimization-based editors like CLIPNeRF. The work highlights a pathway to transfer rich 2D semantic priors into 3D NeRF representations, expanding editable neural graphics of real-world scenes, while noting limitations tied to teacher quality and geometry noise.

Abstract

Emerging neural radiance fields (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations. However, editing a scene represented by a NeRF is challenging, as the underlying connectionist representations such as MLPs or voxel grids are not object-centric or compositional. In particular, it has been difficult to selectively edit specific regions or objects. In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 3D scenes. We propose to distill the knowledge of off-the-shelf, self-supervised 2D image feature extractors such as CLIP-LSeg or DINO into a 3D feature field optimized in parallel to the radiance field. Given a user-specified query of various modalities such as text, an image patch, or a point-and-click selection, 3D feature fields semantically decompose 3D space without the need for re-training and enable us to semantically select and edit regions in the radiance field. Our experiments validate that the distilled feature fields (DFFs) can transfer recent progress in 2D vision and language foundation models to 3D scene representations, enabling convincing 3D segmentation and selective editing of emerging neural graphics representations.
Paper Structure (27 sections, 6 equations, 13 figures, 3 tables)

This paper contains 27 sections, 6 equations, 13 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Implicit Representations.
  4. Geometric Decomposition of Neural Scene Representations
  5. Zero-shot Semantic Segmentation.
  6. Preliminaries
  7. Neural Radiance Fields (NeRF)
  8. Pre-trained Models and Zero-shot Segmentation of Images
  9. Distilled Feature Fields
  10. Distilling Foundation Modules into 3D Feature Fields via Volume Rendering
  11. Query-based Decomposition and Editing
  12. Experiments
  13. 3D Semantic Segmentation
  14. Semantic Segmentation Results.
  15. Impact of Sampling on Semantic Segmentation.
  16. ...and 12 more sections

Figures (13)

  • Figure 1: Left: A Distilled Feature Field (DFF) maps a coordinate $\mathbf{x}$ and a viewing direction $\mathbf{d}$ to density $\sigma$, color $\mathbf{c}$, and feature $\mathbf{f}$. It is trained by minimizing the difference between rendered features and features as predicted by a pre-trained image feature encoder, as well as the rendered color and ground-truth pixel color. Right: At test time, we may decompose and edit 3D space via selecting and manipulating different 3D regions with a variety of queries.
  • Figure 2: Comparison of predictions by coarse and fine MLPs.
  • Figure 3: Appearance edits of specific objects via different query modalities: an image patch or text.
  • Figure 4: Extraction and deletion of specific objects via different query modalities, an image patch or text. The edited views are 3D consistent, unlike an image inpainting baseline suvorov2022lamaresolution
  • Figure 5: Appearance edits of specific objects, compared with decomposition using features of a NeRF hidden layer. For reference, we also show PCA-based visualizations of the features.
  • ...and 8 more figures