RustNeRF: Robust Neural Radiance Field with Low-Quality Images

TL;DR

RustNeRF tackles the challenge of reconstructing robust Neural Radiance Fields from degraded real-world images by introducing a 3D-aware restoration network that leverages multi-view context to restore frames used for NeRF supervision. It couples this restoration with implicit multi-view guidance to exploit redundancy across views and with a quadtree-based sampling scheme to keep training efficient. The approach demonstrates significant gains over baselines on Blender and LLFF under real-world degradations and validates each component through ablations. Overall, RustNeRF enables robust, high-fidelity novel-view synthesis in realistic conditions, expanding the practical applicability of NeRF for AR/VR and other real-world scenarios.

Abstract

Recent work on Neural Radiance Fields (NeRF) exploits multi-view 3D consistency, achieving impressive results in 3D scene modeling and high-fidelity novel-view synthesis. However, there are limitations. First, existing methods assume enough high-quality images are available for training the NeRF model, ignoring real-world image degradation. Second, previous methods struggle with ambiguity in the training set due to unmodeled inconsistencies among different views. In this work, we present RustNeRF for real-world high-quality NeRF. To improve NeRF's robustness under real-world inputs, we train a 3D-aware preprocessing network that incorporates real-world degradation modeling. We propose a novel implicit multi-view guidance to address information loss during image degradation and restoration. Extensive experiments demonstrate RustNeRF's advantages over existing approaches under real-world degradation. The code will be released.

Figures (8)

• Figure 1: We present RustNeRF, a robust NeRF framework that can handle the degraded low-quality images. Traditional NeRF frameworks fail when encounter input views that are degraded for various reasons, and cannot get rid of artifacts when trained with these degraded images, while our RustNeRF can render high-fidelity results.
• Figure 2: An overview of RustNeRF. Input views are first restored via restoration network. The restored views are used for supervising the training process of NeRF model. We further adopt implicit multi-view guidance to enhance the details via excavating the redundant information in multiple views.
• Figure 3: In (a) traditional NeRF training procedure, only the center of each pixel(Real Pixel) will be sampled, thus the whole pixel is supervised by such single inaccurate value. We propose (b) to cast multiple rays(Pseudo Pixels) inside the pixel and calculate the pixel value via weighted sum so as to excavate the supervision signal from other views. The pseudo pixel and real pixel are correspond to the same point in the scene, which illustrates our insight. Note that the distribution of rays in the figure is exaggerated.
• Figure 4: Qualitative results comparison of our method with baseline method. These scenes are taken from Chair(Blender), Hotdog(Blender), Fortress(LLFF) and Room(LLFF) respectively.
• Figure 5: Preprocessing results for trex in LLFF with different preprocessing methods.