NeRF-Det++: Incorporating Semantic Cues and Perspective-aware Depth Supervision for Indoor Multi-View 3D Detection

TL;DR

NeRF-Det++ addresses three key shortcomings of NeRF-Det—semantic ambiguity, inappropriate sampling, and underutilized depth supervision—by introducing Semantic Enhancement, Perspective-aware Sampling, and Ordinal Residual Depth Supervision. The method integrates semantic supervision through 2D semantic maps, biases depth sampling toward near views with Logarithmic and Linear Increment Sampling, and reframes depth estimation as ordinal bin classification plus residual regression, all within a two-branch detection and rendering framework. Empirical results on ScanNetV2 and ARKITScenes show consistent improvements over NeRF-Det, with ablations validating the contributions of each component. The work advances indoor multi-view 3D detection by leveraging NeRF-derived geometry and enhanced supervision to achieve higher accuracy and robustness in complex scenes.

Abstract

NeRF-Det has achieved impressive performance in indoor multi-view 3D detection by innovatively utilizing NeRF to enhance representation learning. Despite its notable performance, we uncover three decisive shortcomings in its current design, including semantic ambiguity, inappropriate sampling, and insufficient utilization of depth supervision. To combat the aforementioned problems, we present three corresponding solutions: 1) Semantic Enhancement. We project the freely available 3D segmentation annotations onto the 2D plane and leverage the corresponding 2D semantic maps as the supervision signal, significantly enhancing the semantic awareness of multi-view detectors. 2) Perspective-aware Sampling. Instead of employing the uniform sampling strategy, we put forward the perspective-aware sampling policy that samples densely near the camera while sparsely in the distance, more effectively collecting the valuable geometric clues. 3)Ordinal Residual Depth Supervision. As opposed to directly regressing the depth values that are difficult to optimize, we divide the depth range of each scene into a fixed number of ordinal bins and reformulate the depth prediction as the combination of the classification of depth bins as well as the regression of the residual depth values, thereby benefiting the depth learning process. The resulting algorithm, NeRF-Det++, has exhibited appealing performance in the ScanNetV2 and ARKITScenes datasets. Notably, in ScanNetV2, NeRF-Det++ outperforms the competitive NeRF-Det by +1.9% in mAP@0.25 and +3.5% in mAP@0.50$. The code will be publicly at https://github.com/mrsempress/NeRF-Detplusplus.

Figures (3)

• Figure 1: The shortcomings of NeRF-Det and our corresponding solutions. Fig. (a) shows the problem of semantic ambiguity. We introduce the Semantic Enhancement module that leverages semantic supervision to enhance the categorical awareness of the detector. Fig. (b) illustrates the limitations of the inappropriate sampling strategy and the insufficient depth supervision. We propose the novel Perspective-aware Sampling, which focuses more on the near, deviating from the conventional uniform sampling approach. Allowing different perspectives to focus on the objects that deserve more attention makes indoor multi-view 3D detection more effective. For example, we can improve the learning of object $B$ from View $2$ while allowing View $1$ to allocate more attention to its nearby objects. Furthermore, instead of directly regressing the original depth values that are hard to optimize, we propose the Ordinal Residual Depth Supervision. It comprises the classification of the ordinal depth bins and the regression of the residual depth values, which is conducive to more stable depth learning.
• Figure 2: Schematic overview of our NeRF-Det++. The framework comprises two branches, i.e., detection and neural rendering. For the detection side, given the multi-view images, we utilize the 2D backbone to extract discriminative features. The camera pose of each image is used to back-project these 2D features to the 3D space, producing the 3D feature grid. As for the rendering branch, we design the Perspective-aware Sampling policy to concentrate on sampling points of more prominent regions. Two MLPs, i.e., the $\Phi_\text{G}$ and $\Phi_\text{C}$, are employed to estimate the volumetric density and color of the sampled points, respectively. To enhance the semantic awareness of the detection features, we introduce the Semantic Enhancement module $\Phi_\text{S}$, which applies semantic supervision to the 2D views. The $\Phi_\text{G}$ is shared and used to produce the opacity field multiplied by the 3D feature grid to generate the geometry-enhanced features. Ultimately, the reinforced features are fed to the detection head to yield the detection outputs.
• Figure 3: Visual comparison between NeRF-Det and NeRF-Det++. Note that our approach only takes posed RGB images as input. The reconstructed mesh is only used for visualization.