Inpaint360GS: Efficient Object-Aware 3D Inpainting via Gaussian Splatting for 360° Scenes

TL;DR

Inpaint360GS presents an object-aware 3D inpainting framework for $360^rac{ ext{o}}{} frac{ ext{deg}}{}$ scenes based on 3D Gaussian Splatting, addressing occlusion and multi-object editing by distilling 2D segmentation into a unified 3D Gaussian field and leveraging virtual camera views for contextual guidance. The method introduces a Key Object Management System and GS-IoU for robust multi-view mask-to-Gaussian associations, followed by efficient per-Gaussian object ID distillation, depth-guided depth/color initialization, and a recursive, depth-informed inpainting pipeline that fuses 2D and 3D objectives through a 3D hybrid supervision. A new 360° inpainting dataset is provided to enable quantitative evaluation across indoor/outdoor scenes with single and multiple occluded objects. Experimental results show state-of-the-art performance in both objective metrics (PSNR, SSIM, LPIPS, FID) and visual fidelity, with faster inpainting and better cross-view consistency than existing baselines. The work advances practical, interactive 360° scene editing by combining explicit 3D representations, robust object-ID propagation, and context-aware virtual-view guidance, while also highlighting remaining challenges such as shadow artifacts and complex texture inpainting.

Abstract

Despite recent advances in single-object front-facing inpainting using NeRF and 3D Gaussian Splatting (3DGS), inpainting in complex 360° scenes remains largely underexplored. This is primarily due to three key challenges: (i) identifying target objects in the 3D field of 360° environments, (ii) dealing with severe occlusions in multi-object scenes, which makes it hard to define regions to inpaint, and (iii) maintaining consistent and high-quality appearance across views effectively. To tackle these challenges, we propose Inpaint360GS, a flexible 360° editing framework based on 3DGS that supports multi-object removal and high-fidelity inpainting in 3D space. By distilling 2D segmentation into 3D and leveraging virtual camera views for contextual guidance, our method enables accurate object-level editing and consistent scene completion. We further introduce a new dataset tailored for 360° inpainting, addressing the lack of ground truth object-free scenes. Experiments demonstrate that Inpaint360GS outperforms existing baselines and achieves state-of-the-art performance. Project page: https://dfki-av.github.io/inpaint360gs/

Figures (35)

• Figure 1: We propose a novel object-aware 3D inpainting method, Inpaint360GS, which flexibly enables object removal and inpainting in 360° scenes. Our approach effectively handles occlusions in multi-object environments and achieves better geometric and appearance consistency compared to existing state-of-the-art methods, including SPIn-NeRF spinnerf, GScream gscream, AuraFusion360 aurafusion360, and GauGroup gaussiangrouping.
• Figure 2: Multi-View Segmentation Comparison. Compared to GauGroup gaussiangrouping our method has more consistent segmentation results across different views.
• Figure 3: Projection of 3D Gaussians onto 2D Segmentation. K-Means algorithm is employed to effectively distinguish between the foreground (i.e., target object) and background Gaussian points.
• Figure 4: Inpaint360GS Architecture Overview. Our framework takes a sequence of RGB images to construct a Gaussian Radiance Field (GRF) and extract per-view object masks using a 2D segmentation foundation model. By associating these masks across views within the GRF, we obtain multi-view consistent object masks and embed them into the Gaussian representation, assigning each Gaussian an object ID. This object-aware GRF enables direct 3D object manipulation, such as click-based or prompt-based removal. After removing target objects, we render at novel camera poses to obtain virtual views $\mathcal{V}$. During 2D inpainting, we recursively perform conditional RGB and depth inpainting, which is then used for depth-guided 3D inpainting.
• Figure 5: Depth Completion. Leveraging the inherent structure of the scene, our method performs depth inpainting without requiring explicit depth alignment.