Few-shot NeRF by Adaptive Rendering Loss Regularization

TL;DR

This work reveals that there exists an inconsistency between the frequency regularization of PE and rendering loss, which prevents few-shot NeRF from synthesizing higher-quality novel views, and proposes Adaptive Rendering loss regularization for few-shot NeRF, dubbed AR-NeRF.

Abstract

Novel view synthesis with sparse inputs poses great challenges to Neural Radiance Field (NeRF). Recent works demonstrate that the frequency regularization of Positional Encoding (PE) can achieve promising results for few-shot NeRF. In this work, we reveal that there exists an inconsistency between the frequency regularization of PE and rendering loss. This prevents few-shot NeRF from synthesizing higher-quality novel views. To mitigate this inconsistency, we propose Adaptive Rendering loss regularization for few-shot NeRF, dubbed AR-NeRF. Specifically, we present a two-phase rendering supervision and an adaptive rendering loss weight learning strategy to align the frequency relationship between PE and 2D-pixel supervision. In this way, AR-NeRF can learn global structures better in the early training phase and adaptively learn local details throughout the training process. Extensive experiments show that our AR-NeRF achieves state-of-the-art performance on different datasets, including object-level and complex scenes.

Figures (7)

• Figure 1: Illustration of the relationship between Positional Encoding (PE) and rendering. We show the reconstruction and generalization performance of different methods under 3 input views. For FreeNeRF yang2023freenerf and our method, AR-NeRF, 10% iterations mean low-frequency PEs are enabled while 100% iterations mean PEs with all frequencies are enabled. The red boxes show low-frequency global structures while the blue boxes show high-frequency local details.
• Figure 2: Adaptive rendering loss weight learning at different iterations. High-frequency PEs are gradually input during training. The larger the color value, the greater the weight. Low-frequency global structures (e.g., roofs) have greater weights in the early training phase. In contrast, the weights of high-frequency local details (e.g., windows) are smaller. When high-frequency PEs are gradually input, the weights of global structures remain high while the weights of local details become greater and greater.
• Figure 3: Overview of AR-NeRF. Our method follows the training pipeline described in \ref{['sec:intro']}. Besides the frequency regularization of PE, we propose adaptive rendering loss regularization. This aligns the frequency relationship between PE and pixel supervision. See \ref{['sec:method']} for more details.
• Figure 4: Qualitative comparison on DTU. We show novel views rendered by different methods in 3 input-view setting.
• Figure 5: Qualitative comparison on LLFF. We show novel views rendered by different methods in 3 input-view setting.