GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding

TL;DR

GP-NeRF introduces a unified framework that merges NeRF with context-aware segmentation by jointly learning radiance and semantic-embedding fields via Field Aggregation Transformer and Ray Aggregation Transformer. It renders semantic-aware features in novel views and uses a 2D segmenter to obtain context-informed predictions, guided by two self-distillation losses: a 2D semantic distill and a depth-guided semantic distill to enforce semantic discrimination and geometric consistency. The approach yields significant improvements on Replica and ScanNet for generalized and finetuned semantic segmentation as well as instance segmentation, while also enhancing reconstruction quality due to semantic-to-radiance consistency. This framework enables context-aware 3D scene understanding with competitive inference efficiency and demonstrates strong potential for downstream perception tasks in real-world scenes.

Abstract

Applying NeRF to downstream perception tasks for scene understanding and representation is becoming increasingly popular. Most existing methods treat semantic prediction as an additional rendering task, \textit{i.e.}, the "label rendering" task, to build semantic NeRFs. However, by rendering semantic/instance labels per pixel without considering the contextual information of the rendered image, these methods usually suffer from unclear boundary segmentation and abnormal segmentation of pixels within an object. To solve this problem, we propose Generalized Perception NeRF (GP-NeRF), a novel pipeline that makes the widely used segmentation model and NeRF work compatibly under a unified framework, for facilitating context-aware 3D scene perception. To accomplish this goal, we introduce transformers to aggregate radiance as well as semantic embedding fields jointly for novel views and facilitate the joint volumetric rendering of both fields. In addition, we propose two self-distillation mechanisms, i.e., the Semantic Distill Loss and the Depth-Guided Semantic Distill Loss, to enhance the discrimination and quality of the semantic field and the maintenance of geometric consistency. In evaluation, we conduct experimental comparisons under two perception tasks (\textit{i.e.} semantic and instance segmentation) using both synthetic and real-world datasets. Notably, our method outperforms SOTA approaches by 6.94\%, 11.76\%, and 8.47\% on generalized semantic segmentation, finetuning semantic segmentation, and instance segmentation, respectively.

Figures (9)

• Figure 1: Our method, called GP-NeRF, achieves remarkable performance improvements for instance and semantic segmentation in both synthesis straub2019replica and real-world dai2017scannet datasets, as shown in the right column of the figure. Here we showcase generalized semantic segmentation, finetuning semantic segmentation, and instance segmentation) with their corresponding reconstruction results. For the left column, the qualitative results of the visualization are presented, showing the effectiveness of our method for simultaneous segmentation and reconstruction. What's more, we visualize our rendered features via PCA in the novel view, demonstrating our method possesses the capability to produce semantic-aware features that can distinguish between different classes and objects.
• Figure 2: Overview of proposed GP-NeRF. Given reference views with their poses, we embed NeRF into the segmenter to perform context-aware semantic $Y_{sem}$ /instance $Y_{ins}$ segmentation and ray reconstruction $Y_{rgb}$ in novel view (Sec. \ref{['sec:framework']}). In detail, we use Transformers to co-aggregate Radiance as well as Semantic-Embedding fields and render them jointly in novel views (Sec. \ref{['subsec:fields']}). Specifically, we propose two self-distillation mechanisms to boost the discrimination and quality of the semantic embedding field (Sec. \ref{['subsec:optimization']}).
• Figure 3: Illustration of training(a) and rendering(b) procedure, where S.E. field denotes Semantic-Embedding Field.
• Figure 4: 2D Semantic Distillation$\mathcal{L}_{\text{S.D}}$ and Depth-Guided Semantic Optimization$\mathcal{L}_{\text{D.G}}$. This figure demonstrates a single raw of our semantic-embedding field. the network "cheat" by rendering all points $\boldsymbol{f}^{sem}_i$ to the same prediction to satisfy $\mathcal{L}_{\text{S.D}}$ supervision. By performing spatial-wise semantic supervision, $\mathcal{L}_{\text{S.D}}$ is able to mitigate the issue of "cheating".
• Figure 5: Semantic quality comparison in Replicastraub2019replica. On the left, we show the rendering results of S-Ray liu2023semanticray and GP-NeRF(ours) in generalized and finetuning settings. On the right, we visualize the PCA results of our rendered semantic features in novel views.