Coherent and Multi-modality Image Inpainting via Latent Space Optimization

TL;DR

PILOT tackles local inpainting in diffusion models by optimizing latent vectors during the reverse diffusion process to enforce background preservation and semantic centralization with respect to multi-modal prompts. It introduces a two-stage pipeline comprising real-time latent optimization and a subsequent latent blending stage, controlled by a coherence scale to balance speed and quality. The approach is compatible with multiple adapters (e.g., ControlNet, DreamBooth) and demonstrates superior coherence, diversity, and prompt fidelity across text, image prompts, scribbles, and subject-driven edits, validated on COCO with human evaluation. Overall, PILOT provides a practical, tuning-free framework that significantly improves coherence and prompt alignment in multimodal inpainting tasks, with broad applicability and controllability for real-world editing scenarios.

Abstract

With the advancements in denoising diffusion probabilistic models (DDPMs), image inpainting has significantly evolved from merely filling information based on nearby regions to generating content conditioned on various prompts such as text, exemplar images, and sketches. However, existing methods, such as model fine-tuning and simple concatenation of latent vectors, often result in generation failures due to overfitting and inconsistency between the inpainted region and the background. In this paper, we argue that the current large diffusion models are sufficiently powerful to generate realistic images without further tuning. Hence, we introduce PILOT (in\textbf{P}ainting v\textbf{I}a \textbf{L}atent \textbf{O}p\textbf{T}imization), an optimization approach grounded on a novel \textit{semantic centralization} and \textit{background preservation loss}. Our method searches latent spaces capable of generating inpainted regions that exhibit high fidelity to user-provided prompts while maintaining coherence with the background. Furthermore, we propose a strategy to balance optimization expense and image quality, significantly enhancing generation efficiency. Our method seamlessly integrates with any pre-trained model, including ControlNet and DreamBooth, making it suitable for deployment in multi-modal editing tools. Our qualitative and quantitative evaluations demonstrate that PILOT outperforms existing approaches by generating more coherent, diverse, and faithful inpainted regions in response to provided prompts.

Figures (16)

• Figure 1: We showcase the versatility of our approach with single and multiple modalities by generating images in various settings, such as text, text $+$ image prompt, text $+$ scribbles, and subject as reference.
• Figure 2: Framework of our PILOT. (a) First we apply our latent optimization strategy, where we adjust the direction of gradients and identify the optimal latent vector every $\tau$ steps, followed by the normal reverse diffusion process. After that, we apply the latent blending strategy until the denoising process is complete. (b) We depict how we optimize the latent vector with prompts from one or more modalities as conditions.
• Figure 3: Qualitative comparison on text-guided image inpainting.
• Figure 4: Qualitative comparison on text guided inpainting with spatial controls
• Figure 5: Ablation study on the coherence scale $\gamma$.