FreeInpaint: Tuning-free Prompt Alignment and Visual Rationality Enhancement in Image Inpainting

TL;DR

<1> FreeInpaint tackles the twin challenges of prompt alignment and visual rationality in text-guided image inpainting by introducing a tuning-free, inference-time framework that directly optimizes diffusion latents. <2> It combines PriNo, which steers the initial noise to align attention with the masked region, and DeGu, a decomposed, reward-driven guidance that balances text alignment, visual coherence, and human preference without retraining. <3> Across multiple diffusion backbones and benchmarks (EditBench and MSCOCO), FreeInpaint consistently improves both prompt adherence and image quality, outperforming training-based and training-free baselines. <4> The approach enables practical, robust inpainting with high fidelity to prompts, while acknowledging limitations for extremely small masks and potential biases from reward models.

Abstract

Text-guided image inpainting endeavors to generate new content within specified regions of images using textual prompts from users. The primary challenge is to accurately align the inpainted areas with the user-provided prompts while maintaining a high degree of visual fidelity. While existing inpainting methods have produced visually convincing results by leveraging the pre-trained text-to-image diffusion models, they still struggle to uphold both prompt alignment and visual rationality simultaneously. In this work, we introduce FreeInpaint, a plug-and-play tuning-free approach that directly optimizes the diffusion latents on the fly during inference to improve the faithfulness of the generated images. Technically, we introduce a prior-guided noise optimization method that steers model attention towards valid inpainting regions by optimizing the initial noise. Furthermore, we meticulously design a composite guidance objective tailored specifically for the inpainting task. This objective efficiently directs the denoising process, enhancing prompt alignment and visual rationality by optimizing intermediate latents at each step. Through extensive experiments involving various inpainting diffusion models and evaluation metrics, we demonstrate the effectiveness and robustness of our proposed FreeInpaint.

Figures (12)

• Figure 1: Comparisons between our FreeInpaint and existing methods. FreeInpaint simultaneously enhances prompt alignment and visual rationality. Zoom in for better view.
• Figure 2: Our tuning-free FreeInpaint framework consists of two key stages: (1) Prior-Guided Noise Optimization (Sec. \ref{['sec:NoiseOpt']}), which optimizes the initial noise $z_T$ at the first denoising step $t=t_\text{ini}$ to concentrate cross-attention ($A^c_{t=t_\text{ini}}$) and self-attention ($A^s_{t=t_\text{ini}}$) maps within the masked region, improving prompt alignment; and (2) Decomposed Training-Free Guidance (Sec. \ref{['sec:guidance']}), which decomposes the conditional distribution of inpainting and leverages text alignment $r_c$, visual rationality $r_m$, and human preference $r_q$ reward models to guide the predicted noise $\epsilon_\theta(z_t, t, c, z^m, M^\prime)$ during denoising, enhancing visual rationality.
• Figure 3: Visualization of cross-attention (cols 3-4) and self-attention (cols 5-6). Row 2 shows misdirected attention causing an unaligned result, while row 3 shows our optimized noise concentrates attention for a successful alignment. The similarity between the first-step ($t=t_\text{ini}$) and averaged maps validates our efficient optimization.
• Figure 4: Sec. \ref{['sec:NoiseOpt']} PriNo improves prompt alignment and Sec. \ref{['sec:guidance']} DeGu further refines image details.
• Figure 5: Comparison against SDI-based approaches.