3D Reconstruction and New View Synthesis of Indoor Environments based on a Dual Neural Radiance Field

TL;DR

The paper tackles indoor 3D reconstruction and novel view synthesis by introducing Du-NeRF, a dual-field NeRF architecture with a SDF-based geometry branch and a density-based rendering branch. A self-supervised view-independent color component guides the learning of geometry, reducing shape-radiance ambiguity and enabling mutual improvements between geometry and rendering. Experiments on NeuralRGBD, Replica, and ScanNet show state-of-the-art performance in both reconstruction quality and view synthesis, with strong results on fine geometry areas. The approach offers a practical pipeline for accurate indoor scene modeling and rendering, with future work including few-shot settings and alternative feature representations.

Abstract

Simultaneously achieving 3D reconstruction and new view synthesis for indoor environments has widespread applications but is technically very challenging. State-of-the-art methods based on implicit neural functions can achieve excellent 3D reconstruction results, but their performances on new view synthesis can be unsatisfactory. The exciting development of neural radiance field (NeRF) has revolutionized new view synthesis, however, NeRF-based models can fail to reconstruct clean geometric surfaces. We have developed a dual neural radiance field (Du-NeRF) to simultaneously achieve high-quality geometry reconstruction and view rendering. Du-NeRF contains two geometric fields, one derived from the SDF field to facilitate geometric reconstruction and the other derived from the density field to boost new view synthesis. One of the innovative features of Du-NeRF is that it decouples a view-independent component from the density field and uses it as a label to supervise the learning process of the SDF field. This reduces shape-radiance ambiguity and enables geometry and color to benefit from each other during the learning process. Extensive experiments demonstrate that Du-NeRF can significantly improve the performance of novel view synthesis and 3D reconstruction for indoor environments and it is particularly effective in constructing areas containing fine geometries that do not obey multi-view color consistency.

Figures (7)

• Figure 1: Dual Neural Radiance Field (Du-NeRF). In (a) Scene Representation, we use a four-layer multi-resolution grid to store geometric features $f_{gi}$, and a hash-based multi-resolution grid for the color features $f_{ci}$ . $f_{gi}$ is decoded to SDF $\phi$ and density $\sigma$ by different MLPs in (b) SDF branch and (c) the Density branch. We provide depth constraints for $\phi$ and introduce additional regularization terms to ensure the stability of its training. We design a depth alignment loss for $\sigma$ to align the two geometry fields. In (d), for color calculation, $\phi$ and $\sigma$ from (b) and (c) are integrated with decoupled view-independent colors to compute the self-supervised loss. The final rendering color sums the view-dependent and view-independent colors to be integrated with the $\sigma$ weights.
• Figure 2: The qualitative reconstruction results on NeuralRGBD datasets. The proposed method can fill in the missing part (highlighted in yellow boxes) and produce smoother planes and clear edges (highlighted in green boxes)
• Figure 3: Qualitative comparison of novel view synthesis results of NeuralRGBD dataset. It can be seen from the results that our method has better visual rendering effects, whether they are striped structures or text on books.
• Figure 4: Qualitative results on ScanNet scenes demonstrate the superior rendering quality of our approach compared to previous NeRF-based methods, especially for images exhibiting severe motion blur, such as the text on a poster (column 3) and the stripes on the floor (column 4).
• Figure 5: Qualitative mesh reconstruction on Scannet scenes. Our method produces visually smoother and cleaner meshes compared to previous methods, as demonstrated by the zoomed-in details provided for comparison.