MS-NeRF: Multi-Space Neural Radiance Fields

TL;DR

MS-NeRF introduces a multi-space decomposition of the scene into virtual sub-spaces to address mirror-like reflections in Neural Radiance Fields. A lightweight MS module replaces the output layer, enabling per-subspace densities and features (or colors) that are fused via a learned gate, compatible with both MLP-based and grid-based NeRF backbones. The method delivers substantial PSNR gains on reflective regions (e.g., up to 4.15 dB on Mip-NeRF 360 with ~0.5% extra parameters and 2.71 dB on TensoRF) without degrading non-mirror regions, and is validated on a newly crafted 33-synthetic + 7-real 360-degree dataset with challenging camera paths. The results demonstrate improved handling of complex light transport through mirrors and refractions, with robust performance across architectures and various supervision signals, and the authors provide code and data for reproducibility.

Abstract

Existing Neural Radiance Fields (NeRF) methods suffer from the existence of reflective objects, often resulting in blurry or distorted rendering. Instead of calculating a single radiance field, we propose a multi-space neural radiance field (MS-NeRF) that represents the scene using a group of feature fields in parallel sub-spaces, which leads to a better understanding of the neural network toward the existence of reflective and refractive objects. Our multi-space scheme works as an enhancement to existing NeRF methods, with only small computational overheads needed for training and inferring the extra-space outputs. We design different multi-space modules for representative MLP-based and grid-based NeRF methods, which improve Mip-NeRF 360 by 4.15 dB in PSNR with 0.5% extra parameters and further improve TensoRF by 2.71 dB with 0.046% extra parameters on reflective regions without degrading the rendering quality on other regions. We further construct a novel dataset consisting of 33 synthetic scenes and 7 real captured scenes with complex reflection and refraction, where we design complex camera paths to fully benchmark the robustness of NeRF-based methods. Extensive experiments show that our approach significantly outperforms the existing single-space NeRF methods for rendering high-quality scenes concerned with complex light paths through mirror-like objects. The source code, dataset, and results are available via our project page: https://zx-yin.github.io/msnerf/.

Figures (23)

• Figure 1: These are test views from the novel mirror-passing-through path. The first row is in front of the mirror, while the last row is behind the mirror.
• Figure 2: The virtual image created by the mirror is visible only in a small range of views, which violates the multi-view consistency.
• Figure 3: The first row is training view examples in the two scenes. In scene A, there is only a plant in front of a mirror, while in scene B we carefully place another plant to match the exact position where the virtual image lies. The second row is test views with rendered depth from the vanilla NeRF trained on the toy scenes. As demonstrated, NeRF can avoid the trap of treating reflected images as textures when the 'virtual image' satisfies multi-view consistency.
• Figure 4: We visualize composed RGB and depth maps of novel views and the decoded images with the corresponding weights and depth maps of all sub-spaces from our $\textrm{MS-NeRF}_B$ model in Sec. \ref{['sec:main_res']}. The results show that our method successfully decomposes virtual images into certain sub-spaces.
• Figure 5: (a) render result by $\textrm{MS-NeRF}_B$. (b) visualization of depth maps rendered by $\textrm{MS-NeRF}_B$. (c) render result by NeRF. (d) visualization of depth maps rendered by NeRF. The visualization from Sec. \ref{['sec:main_res']} indicates that our MS module understands the light transport at the occurrence of reflections, and the common parts can also be rendered correctly.