RaFE: Generative Radiance Fields Restoration

TL;DR

RaFE tackles the challenge of reconstructing high-quality radiance fields from degraded multi-view images by first applying off-the-shelf 2D restoration to each view and then learning a generative NeRF with a two-level tri-plane representation: a fixed coarse tri-plane $P_c$ derived from low-quality inputs and a learnable fine residual tri-plane $P_f$ produced by a GAN, forming a distribution over high-quality NeRFs $p_r(\boldsymbol{\phi}|I_h)$. Training optimizes adversarial and perceptual objectives on rendered patches to align with restored images while preserving geometry, and employs a patch-based strategy to stabilize learning. Across synthetic and real degradations, RaFE achieves superior perceptual rendering quality and refined geometry, enabling realistic, diverse 3D reconstructions from degraded inputs.

Abstract

NeRF (Neural Radiance Fields) has demonstrated tremendous potential in novel view synthesis and 3D reconstruction, but its performance is sensitive to input image quality, which struggles to achieve high-fidelity rendering when provided with low-quality sparse input viewpoints. Previous methods for NeRF restoration are tailored for specific degradation type, ignoring the generality of restoration. To overcome this limitation, we propose a generic radiance fields restoration pipeline, named RaFE, which applies to various types of degradations, such as low resolution, blurriness, noise, compression artifacts, or their combinations. Our approach leverages the success of off-the-shelf 2D restoration methods to recover the multi-view images individually. Instead of reconstructing a blurred NeRF by averaging inconsistencies, we introduce a novel approach using Generative Adversarial Networks (GANs) for NeRF generation to better accommodate the geometric and appearance inconsistencies present in the multi-view images. Specifically, we adopt a two-level tri-plane architecture, where the coarse level remains fixed to represent the low-quality NeRF, and a fine-level residual tri-plane to be added to the coarse level is modeled as a distribution with GAN to capture potential variations in restoration. We validate RaFE on both synthetic and real cases for various restoration tasks, demonstrating superior performance in both quantitative and qualitative evaluations, surpassing other 3D restoration methods specific to single task. Please see our project website https://zkaiwu.github.io/RaFE-Project/.

Figures (8)

• Figure 1: High-quality restoration of radiance fields from various types of degradation. Given only degraded images, our method can restore high-quality NeRF. It is a generic approach that can be applied to various types of degradations, resulting in refinement of both geometry and appearance.
• Figure 2: Overview of our pipeline. Given multi-view degraded images, we utilize the off-the-shelf methods to restore high-quality multi-view images. Then, we train our Generative Restoration NeRF to generate a high-quality scene.
• Figure 3: (a) Error map between ground truth and our method/NeRF-Perframe. (b) we showcase that the visual quality can be much better even with lower PSNR scores.
• Figure 4: Visual results for super-resolution and mixed-degradation tasks. We show that our method is capable of generating detailed geometry and texture while other methods tend to be smooth in both geometry and texture. We recommend zooming in for better visualization.
• Figure 5: (a)Geometry comparisons between NeRF-SWINIR/NeRFLIX and our method. (b)Comparisons between using different 2D restoration models.