Perspective-aware 3D Gaussian Inpainting with Multi-view Consistency

TL;DR

The paper tackles multi-view inconsistencies in 3D Gaussian inpainting by introducing PAInpainter, a perspective-aware pipeline that builds a perspective graph of viewpoints to guide adaptive view sampling, propagates inpainted content as priors across adjacent views, and employs a dual-feature (RGB-depth) verification to enforce cross-view coherence. It alternates between multi-view inpainting and 3D Gaussian optimization, updating the 3D scene using a loss that combines L1 and SSIM terms on inpainted views while excluding masked regions. Empirical results on NeRF Blender, SPIn-NeRF, and NeRFiller demonstrate state-of-the-art performance in PSNR, SSIM, LPIPS, and FID, with a notable PSNR of 29.51 dB on NeRFiller and robust generalization across diverse scenarios. The method avoids diffusion-model fine-tuning, instead leveraging content propagation and consistency verification to achieve high-fidelity, texture-preserving, geometrically coherent 3D reconstructions, with practical implications for AR/VR and holographic multimedia applications.

Abstract

3D Gaussian inpainting, a critical technique for numerous applications in virtual reality and multimedia, has made significant progress with pretrained diffusion models. However, ensuring multi-view consistency, an essential requirement for high-quality inpainting, remains a key challenge. In this work, we present PAInpainter, a novel approach designed to advance 3D Gaussian inpainting by leveraging perspective-aware content propagation and consistency verification across multi-view inpainted images. Our method iteratively refines inpainting and optimizes the 3D Gaussian representation with multiple views adaptively sampled from a perspective graph. By propagating inpainted images as prior information and verifying consistency across neighboring views, PAInpainter substantially enhances global consistency and texture fidelity in restored 3D scenes. Extensive experiments demonstrate the superiority of PAInpainter over existing methods. Our approach achieves superior 3D inpainting quality, with PSNR scores of 26.03 dB and 29.51 dB on the SPIn-NeRF and NeRFiller datasets, respectively, highlighting its effectiveness and generalization capability.

Figures (12)

• Figure 1: Current challenges in 3D Gaussian inpainting: a) the fine-tune based inpainter trained for specific tasks cao2024mvinpainter experiences significant performance decline when applied to general inpainting scenarios; b) the joint-view inpainting method weber2024nerfiller struggles with inconsistency across multi-view images, resulting in noisy inpainting results; c) the DU strategy haque2023instruct leads to texture degradation in both inpainted multi-view images and the final 3D scene.
• Figure 1: The inpaint content propagation between anchor images and corresponding adjacent images. With our perspective graph sampling strategy, the anchor image provides sufficient and accurate prior to adjacent images to guide consistent multi-view inpainting.
• Figure 2: The overall pipeline of our proposed PAInpainter. Based on the constructed perspective graph, our approach iteratively performs multi-view image inpainting and 3D Gaussian training. The adaptive graph sampling algorithm enables efficient inpaint content propagation across adjacent viewpoints, while consistency verification ensures coherent multi-view inpainting results, thereby improving the 3D inpainting quality.
• Figure 2: Visualization for consistency verification. Red contours delineate mask boundaries and green boxes highlight top-scoring candidates selected for 3DGS optimization. The upper-left number of each candidate represents the consistency score. This module reliably identifies inpainted regions exhibiting both textural and geometric consistency (zoom for details), enhancing performance and robustness.
• Figure 3: Overview of PAInpainter for multi-view consistent 3D Gaussian inpainting. Our method is built upon the pretrained SD2 Rombach2022SD2 and incorporates three key components: 1) perspective graph models spatial relationships among cameras to guide adjacent view sampling; 2) inpaint content propagation transmits inpainting content across adjacent views sampled from perspective graph, providing extra visual priors for diffusion inpainting; 3) consistency verification evaluates inpainted results based on texture and geometric features coherence. The perspective-aware graph sampling contributes to effective content propagation and consistency verification across multiple views.