Re-Nerfing: Improving Novel View Synthesis through Novel View Synthesis

TL;DR

This paper proposes Re-Nerfing, a simple and general add-on approach that renders novel views between existing ones and augment the training data to optimize a second model, introducing additional multi-view constraints and allowing the second model to converge to a better solution.

Abstract

Recent neural rendering and reconstruction techniques, such as NeRFs or Gaussian Splatting, have shown remarkable novel view synthesis capabilities but require hundreds of images of the scene from diverse viewpoints to render high-quality novel views. With fewer images available, these methods start to fail since they can no longer correctly triangulate the underlying 3D geometry and converge to a non-optimal solution. These failures can manifest as floaters or blurry renderings in sparsely observed areas of the scene. In this paper, we propose Re-Nerfing, a simple and general add-on approach that leverages novel view synthesis itself to tackle this problem. Using an already trained NVS method, we render novel views between existing ones and augment the training data to optimize a second model. This introduces additional multi-view constraints and allows the second model to converge to a better solution. With Re-Nerfing we achieve significant improvements upon multiple pipelines based on NeRF and Gaussian-Splatting in sparse view settings of the mip-NeRF 360 and LLFF datasets. Notably, Re-Nerfing does not require prior knowledge or extra supervision signals, making it a flexible and practical add-on.

Figures (6)

• Figure 1: Examples of synthesized views by 3DGS kerbl2023gaussiansplat, Instant-NGP muller2022instant, RegNeRF niemeyer2022regnerf, with and without the proposed Re-Nerfing. Our approach improves NVS by improving the optimization, reaching better global solutions thanks to the addition of novel views synthesized by the baseline model to the training data. Image crops show test views from the mip-NeRF 360 barron2022mip360 dataset for 3DGS, iNGP trained on 30 views and from the LLFF mildenhall2019llff dataset for RegNeRF trained on 3 views.
• Figure 2: Re-Nerfing is a multi-stage framework. Compatible with any NVS pipeline, it operates by first training a model with the available views (green, 1st). This model is then used to generate novel views from camera poses to improve the scene coverage (orange). Then, we compute the model's uncertainty on such novel views (blue) and discard the uncertain regions (orange-blue). Finally, we use these masked views and the original ones to train a second model (orange, 2nd). The process can be repeated iteratively by generating the novel views with the second model (3rd stage).
• Figure 3: Qualitative results on cropped images from the test set of the mip-NeRF 360 and LLFF datasets. 3DGS kerbl2023gaussiansplat and Instant-NGP muller2022instant were trained on the mip-NeRF 360 dataset in a 30 view setting, while RegNeRF niemeyer2022regnerf was trained on LLFF with only 3 views. Qualitative results of PyNeRF turki2024pynerf can be found in the supplementary material.
• Figure 4: Qualitative results on cropped images from the test set of the mip-NeRF 360 dataset barron2022mip360 showcasing PyNeRF turki2024pynerf in sparse-view settings, with and without the proposed Re-Nerfing.
• Figure 5: Illustration of the trade-offs arising when synthesizing novel views (orange) in between existing ones (green). The numbers indicate the interpolation factors. The closer to the original views, the higher the quality of the synthesis and the lower the uncertainty. Moving further from the original views (i.e., toward 0.5), the quality degrades and the uncertainty increases. Yet, an opposite trade-off occurs as views too close to the original ones do not bring any extra information. We balance this by using many views and removing artifacts thanks to the uncertainty estimates.