Geometry-Aware Diffusion Models for Multiview Scene Inpainting

TL;DR

The paper tackles multiview 3D scene inpainting where partial views must be completed consistently across viewpoints. It sidesteps the blurriness of radiance-field fusion by fusing cross-view information in a learned diffusion latent space, guided by geometry-aware, reference-based cues. A two-component system—a scene-geometry estimator (DUSt3R) and a geometry-aware inpainting diffusion model—augmented with an autoregressive procedure, enables high-quality, multiview-consistent inpainting even with few views. The approach achieves state-of-the-art results on SPIn-NeRF and NeRFiller, producing sharper and more coherent outputs than 3D radiance-field baselines, while maintaining robust performance in few-view scenarios. This work advances practical 3D scene editing by enabling sharp, consistent inpainting without dense view coverage, broadening applicability to real-world scenes and sparse data.

Abstract

In this paper, we focus on 3D scene inpainting, where parts of an input image set, captured from different viewpoints, are masked out. The main challenge lies in generating plausible image completions that are geometrically consistent across views. Most recent work addresses this challenge by combining generative models with a 3D radiance field to fuse information across a relatively dense set of viewpoints. However, a major drawback of these methods is that they often produce blurry images due to the fusion of inconsistent cross-view images. To avoid blurry inpaintings, we eschew the use of an explicit or implicit radiance field altogether and instead fuse cross-view information in a learned space. In particular, we introduce a geometry-aware conditional generative model, capable of multi-view consistent inpainting using reference-based geometric and appearance cues. A key advantage of our approach over existing methods is its unique ability to inpaint masked scenes with a limited number of views (i.e., few-view inpainting), whereas previous methods require relatively large image sets for their 3D model fitting step. Empirically, we evaluate and compare our scene-centric inpainting method on two datasets, SPIn-NeRF and NeRFiller, which contain images captured at narrow and wide baselines, respectively, and achieve state-of-the-art 3D inpainting performance on both. Additionally, we demonstrate the efficacy of our approach in the few-view setting compared to prior methods.

Figures (16)

• Figure 1: Visualization of our target inpainting tasks. We target three tasks: (i) narrow-baseline object removal, (ii) wide-baseline scene completion and (iii) few-view inpainting. Here, we show examples of our outputs for each task, with the corresponding masked inputs shown in the top left corner of each image.
• Figure 2: Overview of our geometric-aware 3D scene inpainter. (a) A visualization of the reference-based geometric cues. Back-faces are always covered by shadow volumes. $T_i(I_i), F_i, B_i, C_i, \widehat{D}_i$ denote the rendered photometric content, front-face mask, back-face mask, shadow mask, and disparity, respectively, for the reference view $i$. (b) A step-by-step visualization of our autoregressive inpainting process. Note that the scene geometry consists of separate meshes for each image, not a single harmonized mesh, as shown here for simplicity. Here, we are only showing one diffusion step for the geometry-aware inpainting model. $\mathcal{E}, z_t, I, M, A_{\mathcal{R}_i}, \mathcal{G}_{\mathcal{R}_i}$ denote the VAE encoder (which maps an image to the latent space of Stable Diffusion), the diffusion latent at timestep $t$, the masked image latent, the mask, the appearance cues for reference view $i$, and the geometric cues for reference view $i$, respectively; see §\ref{['supp:subsec:autoregressive-qualitative']} for an example of autoregressive steps.
• Figure 3: Qualitative object removal comparisons on the SPIn-NeRF dataset. Notice other methods produce blurry regions (e.g., bench end) due to multiview inconsistencies, while ours preserves sharpness and visual plausibility. Please zoom in for details.
• Figure 4: Qualitative scene completion comparisons on the NeRFiller dataset. NeRFiller can converge to blurry content, due to mixing divergent views, while ours generates and then propagates sharp content (e.g., see details in the backpack or window glass in the zoomed patches). Each view in a scene is denoted by $v_i$, where $i$ is the view index.
• Figure 5: Qualitative comparisons for few-view inpainting. Our inpainted images are sharper and more visually plausible.