SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image

TL;DR

SinNeRF addresses the extreme sparse-view problem by training a neural radiance field from a single reference image of a complex scene. It introduces a semi-supervised framework that uses geometry pseudo labels via image warping and semantic pseudo labels via a patch-based adversarial texture guide and a global semantic prior from a pre-trained DINO-ViT, all within a progressive training regime. Across NeRF-Synthetic, LLFF, and DTU, SinNeRF delivers photo-realistic novel-view synthesis and outperforms state-of-the-art sparse-view baselines without multi-view pretraining. This work broadens NeRF applicability to single-shot captures, though it notes model training efficiency as a limitation and future work area.

Abstract

Despite the rapid development of Neural Radiance Field (NeRF), the necessity of dense covers largely prohibits its wider applications. While several recent works have attempted to address this issue, they either operate with sparse views (yet still, a few of them) or on simple objects/scenes. In this work, we consider a more ambitious task: training neural radiance field, over realistically complex visual scenes, by "looking only once", i.e., using only a single view. To attain this goal, we present a Single View NeRF (SinNeRF) framework consisting of thoughtfully designed semantic and geometry regularizations. Specifically, SinNeRF constructs a semi-supervised learning process, where we introduce and propagate geometry pseudo labels and semantic pseudo labels to guide the progressive training process. Extensive experiments are conducted on complex scene benchmarks, including NeRF synthetic dataset, Local Light Field Fusion dataset, and DTU dataset. We show that even without pre-training on multi-view datasets, SinNeRF can yield photo-realistic novel-view synthesis results. Under the single image setting, SinNeRF significantly outperforms the current state-of-the-art NeRF baselines in all cases. Project page: https://vita-group.github.io/SinNeRF/

Figures (5)

• Figure 1: Given only a single reference view as input, our novel semi-supervised framework effectively trains a neural radiance field. In contrast, previous method deng2021depth shows inconsistent geometry when synthesizing novel views.
• Figure 2: An overview of our SinNeRF, where we synthesize patches from the reference view and unseen views. We train this semi-supervised framework via ground truth color and depth labels of the reference view and pseudo labels on unseen views. We use image warping to obtain geometry pseudo labels and utilize adversarial training as well as a pre-trained ViT for semantic pseudo labels.
• Figure 3: Novel view synthesis results of different methods on NeRF synthetic and LLFF Dataset.
• Figure 4: Novel view synthesis results of different methods on DTU dataset.
• Figure 5: Novel view synthesis from different variants of our proposed model.