Diffusion Models with Anisotropic Gaussian Splatting for Image Inpainting

TL;DR

The paper tackles the challenge of realistic image inpainting, especially for large missing regions where preserving structure and texture is difficult. It combines diffusion-based inpainting with anisotropic Gaussian splatting, where missing regions are modeled using gradient-adaptive Gaussians whose multi-scale splat maps guide the diffusion process. Key contributions include the anisotropic Gaussian modeling, integration of splat guidance into a diffusion inpainting network, multi-scale splatting, and comprehensive experiments showing superior fidelity and structural coherence on CIFAR-10 and CelebA. The approach demonstrates that explicit structural priors can substantially improve inpainting quality, with potential impact on photo editing, restoration, and occlusion handling in vision systems.

Abstract

Image inpainting is a fundamental task in computer vision, aiming to restore missing or corrupted regions in images realistically. While recent deep learning approaches have significantly advanced the state-of-the-art, challenges remain in maintaining structural continuity and generating coherent textures, particularly in large missing areas. Diffusion models have shown promise in generating high-fidelity images but often lack the structural guidance necessary for realistic inpainting. We propose a novel inpainting method that combines diffusion models with anisotropic Gaussian splatting to capture both local structures and global context effectively. By modeling missing regions using anisotropic Gaussian functions that adapt to local image gradients, our approach provides structural guidance to the diffusion-based inpainting network. The Gaussian splat maps are integrated into the diffusion process, enhancing the model's ability to generate high-fidelity and structurally coherent inpainting results. Extensive experiments demonstrate that our method outperforms state-of-the-art techniques, producing visually plausible results with enhanced structural integrity and texture realism.

Figures (5)

• Figure 1: Overview of the proposed inpainting method integrating anisotropic Gaussian splatting with a diffusion-based inpainting network. The anisotropic Gaussian splats model the spatial influence and uncertainty in the missing regions, guiding the diffusion process for more accurate restoration.
• Figure 2: Illustration of anisotropic Gaussian splatting in 2D, representing the spatial influence and uncertainty in missing regions. Each missing pixel is modeled as an anisotropic Gaussian function whose shape adapts to local image structures, allowing for more accurate guidance in the inpainting process.
• Figure 3: Visualization of the Gaussian splat map with adaptive amplitudes. The left image shows the original image, and the right image displays the corresponding 3D representation of the Gaussian splat map.
• Figure 4: The inputs to the inpainting network, including the incomplete image $(I \odot (1 - M))$, the mask $M$, the normalized Gaussian splat map $S_{\text{norm}}$, the edge map $E$, and the attention map $A_{\text{att}}$.
• Figure 5: Qualitative inpainting results on the CelebA dataset (left) and CIFAR-10 dataset (right)