NeRF-MIR: Towards High-Quality Restoration of Masked Images with Neural Radiance Fields

TL;DR

This work addresses masked image restoration in multi-view scenes by introducing NeRF-MIR, a NeRF-based framework that restores occluded regions through a patch-aware ray-emitting strategy (PERE), a self-training progressive restoration process (PIRE), and a dynamically weighted loss that gradually emphasizes masked regions. By rebalancing ray allocation toward textured patches and iteratively refining masked content using multi-view consistency, NeRF-MIR achieves substantial improvements over both 2D inpainting baselines and prior NeRF-based restoration methods. The authors construct three masked datasets (LLFF-M, Spaces-M, Blender-M) and demonstrate that NeRF-MIR yields higher PSNR/SSIM and lower LPIPS, while also producing plausible novel-view synthesis in masked scenarios. Overall, the approach advances 3D-aware restoration with NeRFs and shows practical potential for cleaning distractors in real-world scenes.

Abstract

Neural Radiance Fields (NeRF) have demonstrated remarkable performance in novel view synthesis. However, there is much improvement room on restoring 3D scenes based on NeRF from corrupted images, which are common in natural scene captures and can significantly impact the effectiveness of NeRF. This paper introduces NeRF-MIR, a novel neural rendering approach specifically proposed for the restoration of masked images, demonstrating the potential of NeRF in this domain. Recognizing that randomly emitting rays to pixels in NeRF may not effectively learn intricate image textures, we propose a \textbf{P}atch-based \textbf{E}ntropy for \textbf{R}ay \textbf{E}mitting (\textbf{PERE}) strategy to distribute emitted rays properly. This enables NeRF-MIR to fuse comprehensive information from images of different views. Additionally, we introduce a \textbf{P}rogressively \textbf{I}terative \textbf{RE}storation (\textbf{PIRE}) mechanism to restore the masked regions in a self-training process. Furthermore, we design a dynamically-weighted loss function that automatically recalibrates the loss weights for masked regions. As existing datasets do not support NeRF-based masked image restoration, we construct three masked datasets to simulate corrupted scenarios. Extensive experiments on real data and constructed datasets demonstrate the superiority of NeRF-MIR over its counterparts in masked image restoration.

Figures (15)

• Figure 1: The process of obtaining the mask matrix is illustrated in the first row. Two examples of common image distractors in nature are shown in the second and third rows: (a) the output results of YOLOv5, (b) the masked results based on the center coordinates of bounding boxes, and (c) the reconstructed camera parameters of masked images.
• Figure 2: Sample scenes from our LLFF-M and Spaces-M datasets. The images of Fern (a) and Spaces_073 (d) are first split into small square patches. Then, we randomly or fixedly select patches in each view to mask by different shapes, sizes, and levels, which are shown in (b-c) and (e-f), respectively.
• Figure 3: The framework of our method contains two major components: the Patch-Based Entropy for Ray Emitting (PERE in short) module and the Progressively Iterative Restoration (PIRE in short) module. Input images are masked after patch segmentation, then we evaluate the patch entropy and redistribute rays to each patch as described in Sec. \ref{['sec:entropy']}. We iteratively update the masked regions $\mathcal{P}_k$ by the PIRE mechanism that leverages multi-view information (Sec. \ref{['sec:progressive']}). The process is repeated several times, with the prediction of the prior stage serving as the training data for the next stage.
• Figure 4: The heat map of ray distribution. The areas of smooth texture (R1) and intricate texture (R2) are bounded by a blue box.
• Figure 5: The detailed procedure of PIRE. The masked regions of input images in the $t$-th stage are replaced by the predictions from $\mathcal{P}^{(t-1)}_{mask}$ in the ($t$-1)-th stage. The predictions of each stage for masked-out pixels serve as additional training data for the following next stage.