OPONeRF: One-Point-One NeRF for Robust Neural Rendering

TL;DR

Experimental results show that the OPONeRF outperforms state-of-the-art NeRFs on various evaluation metrics through benchmark experiments and cross-scene evaluations, and the efficacy of the proposed method is shown via experimenting on other existing generalization-based benchmarks and incorporating the idea of One-Point-One NeRF into other advanced baseline methods.

Abstract

In this paper, we propose a One-Point-One NeRF (OPONeRF) framework for robust scene rendering. Existing NeRFs are designed based on a key assumption that the target scene remains unchanged between the training and test time. However, small but unpredictable perturbations such as object movements, light changes and data contaminations broadly exist in real-life 3D scenes, which lead to significantly defective or failed rendering results even for the recent state-of-the-art generalizable methods. To address this, we propose a divide-and-conquer framework in OPONeRF that adaptively responds to local scene variations via personalizing appropriate point-wise parameters, instead of fitting a single set of NeRF parameters that are inactive to test-time unseen changes. Moreover, to explicitly capture the local uncertainty, we decompose the point representation into deterministic mapping and probabilistic inference. In this way, OPONeRF learns the sharable invariance and unsupervisedly models the unexpected scene variations between the training and testing scenes. To validate the effectiveness of the proposed method, we construct benchmarks from both realistic and synthetic data with diverse test-time perturbations including foreground motions, illumination variations and multi-modality noises, which are more challenging than conventional generalization and temporal reconstruction benchmarks. Experimental results show that our OPONeRF outperforms state-of-the-art NeRFs on various evaluation metrics through benchmark experiments and cross-scene evaluations. We further show the efficacy of the proposed method via experimenting on other existing generalization-based benchmarks and incorporating the idea of One-Point-One NeRF into other advanced baseline methods.

Figures (13)

• Figure 1: Qualitative comparison with baseline methods on the benchmarks of test-time scene perturbations. The training scene before perturbation and testing groundtruth (GT) after perturbation belonging to the same camera view are shown. In the top two rows, we trained on the first timeframe and tested on different timeframes of the same task scene. In the bottom row, we trained on the first timeframe and tested on the timeframes of another task scene. Advanced baselines such as MVSNeRF, NeuRay and GeoNeRF produce significantly defective rendering results or even fail when handling small but unpredictable test-time scene perturbations. Notably, GeoNeRF renders all-black results when encountered with foreground movements. OPONeRF better preserves the details on both moving foreground and static background areas.
• Figure 2: Illustration of OPONeRF. To render the querying view, OPONeRF firstly extracts a pair of scene-level features ($\mathbf{F}$ and $\mathbf{A}$) using a geometry encoder. The initial point representation $f_\mathbf{x}$ and adaptiveness factor $a_\mathbf{x}$ are interpolated from $\mathbf{F}$ and $\mathbf{A}$ via querying on $\mathbf{x}$. OPONeRF then parallelly learns a series of PCDs taking $\mathbf{F}$ as input and producing the geometry-aware and layer-variant candidate parameters $\mathbf{W}^l_a$ for the target layers of OPONeRF renderer. For each $\mathbf{x}$, we learn its final probabilistic representation $\mathcal{F}_\mathbf{x}$ and the fused $\mathcal{A}_\mathbf{x}$. The renderer parameters personalized for each $\mathbf{x}$ are adaptively controlled by $\mathcal{A}_\mathbf{x}$ via selecting from candidate parameters. In this way, OPONeRF learns a personalized neural renderer for each sampled $x$. The OPONeRF renderer is a Ray Transformer with layers personalized for each $\mathbf{x}$ as well as shareable ones that are agnostic to $\mathbf{x}$. The output of OPONeRF renderer will be processed by the classical volume rendering to get the properties of final querying view. (Best viewed in color.)
• Figure 3: The parameter candidate decoder (PCD) responsible for the $l$-th target layer parameterized by $\mathbf{W}^l_a$. OPONeRF parallelly learns a series of PCDs each responsible for a unique target layer.
• Figure 4: Probabilistic modeling of point representation.
• Figure 5: Upper: conditioning the parameters of the renderer $\mathcal{M}_\Theta$ on the global representation of the given scene. Lower: learning to personalize a unique neural renderer $\mathcal{M}_\Theta^a(\mathbf{x}_i)$ for each densely sampled coordinate $\mathbf{x}_i$. In this figure, we use 2D image patches instead of 3D local coordinates to represent $\mathbf{x}_i$s for intuitive illustration. (Best viewed in color.)