${M^2D}$NeRF: Multi-Modal Decomposition NeRF with 3D Feature Fields

TL;DR

The paper introduces $M^2D$NeRF, a NeRF-based framework that adds dual visual-language semantic volumes to enable text-based and patch-based 3D edits. It builds these volumes via multi-modal feature distillation from DINO and CLIP-LSeg, and enforces explicit cross-modal alignment with a multi-modal similarity constraint plus a patch-level joint contrastive loss to sharpen object boundaries. Experiments on real LLFF scenes show improved 3D decomposition, segmentation, and localization compared to prior NeRF-based methods, demonstrating robust multi-modal 3D editing within a single model. This approach has practical impact for scene understanding in AR/VR and robotics, enabling more intuitive and accurate 3D manipulations without dense 3D annotations.

Abstract

Neural fields (NeRF) have emerged as a promising approach for representing continuous 3D scenes. Nevertheless, the lack of semantic encoding in NeRFs poses a significant challenge for scene decomposition. To address this challenge, we present a single model, Multi-Modal Decomposition NeRF (${M^2D}$NeRF), that is capable of both text-based and visual patch-based edits. Specifically, we use multi-modal feature distillation to integrate teacher features from pretrained visual and language models into 3D semantic feature volumes, thereby facilitating consistent 3D editing. To enforce consistency between the visual and language features in our 3D feature volumes, we introduce a multi-modal similarity constraint. We also introduce a patch-based joint contrastive loss that helps to encourage object-regions to coalesce in the 3D feature space, resulting in more precise boundaries. Experiments on various real-world scenes show superior performance in 3D scene decomposition tasks compared to prior NeRF-based methods.

Figures (10)

• Figure 1: Multi-Modal Decomposition NeRF (${M^2D}$NeRF). Using DINO and CLIP-LSeg as teacher models for feature distillation, our ${M^2D}$NeRF facilitates multi-modal queries for 3D scene decomposition. By establishing explicit relationships within multi-modal feature fields, our ${M^2D}$NeRF supports tasks with both text and patch-based queries and exhibits sharper and more accurate object identification.
• Figure 2: ${M^2D}$NeRF framework. Left: We expand the NeRF backbone by introducing two additional branches $\varphi_{v}$ and $\varphi_{l}$ ($\varphi_{c}$ denotes the original color branch). Given a 3D sample point $\mathbf{x}$, the three branches produces the volumetric color $\hat{c}(\mathbf{x},\mathbf{d})$ or feature $\hat{f}(\mathbf{x})$ (consisting of the visual $\hat{f}_v(\mathbf{x})$ and language $\hat{f}_l(\mathbf{x})$ features). By considering the volume density $\sigma$, we get the rendered color $\hat{C}(\mathbf{r})$ and visual $\hat{\mathbf{F}}_v(\mathbf{r})$ and language $\hat{\mathbf{F}}_l(\mathbf{r})$ features for each ray. Right: We extract per-pixel multi-modal features $\mathbf{F}_{v}(\mathbf{r})$ and $\mathbf{F}_{l}(\mathbf{r})$ using models $\Phi_{v}$ and $\Phi_{l}$ (visual model DINO dino2021caron and language model CLIP-LSeg lseg2022li in our work). These act as 2D teacher models that supervise the student feature fields via distillation loss ($\mathcal{L}_{distill}$). We build both 3D (multi-modal similarity $\mathcal{L}_{mms}$) and patch-level 2D (joint contrastive learning scheme $\mathcal{L}_{cl}$) relationships between multi-modal features for better scene understanding.
• Figure 3: Patch query-based 3D extraction and color edit. We compare with DFF-DINO and N3F showing two views that highlight the multi-view consistency in 3D object extraction. We recommend zooming in and focusing on the yellow boxes to inspect intricate boundaries and fine details. We can then edit these extracted 3D regions, like changing the color (last column for each view).
• Figure 4: Qualitative results with text query. Our ${M^2D}$NeRF effectively extracts corresponding objects based on text queries (left column) with a fixed threshold, showcasing superior scene understanding. In contrast, DFF-LSeg struggles to find a suitable threshold for precise object extraction. With just tiny changes to the threshold (see blue numbers for thresholds used for DFF-LSeg), DFF-LSeg can end up extracting almost the entire image or nothing (see last rows).
• Figure 5: Segmentation results on {Flower, Fortress} scenes. Compared with NeRF-SOS, our ${M^2D}$NeRF gets more complete objects. Compared with DFF and N3F, our method can greatly reduce the noise. Our results are more close to the ground-truth. We recommend zooming in and focusing on the yellow boxes to inspect details.