Aligned Stable Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency

TL;DR

This work tackles two persistent problems in latent inpainting: unwanted object insertion and color inconsistency. It introduces ASUKA, a post-training framework that uses a context-stable MAE prior to guide masked-region reconstruction and a specialized local harmonization decoder to ensure color continuity, all while preserving the generation capacity of frozen models. ASUKA-I adapts to U-Net–based backbones, and ASUKA-II extends to transformer-based generators with per-layer MAE conditioning, scaled positional encoding, and improved mask handling, yielding more robust corrections. Evaluations on Places2 and the MISATO benchmark demonstrate state-of-the-art suppression of hallucinations and improved color consistency compared with diffusion-based, rectified-flow, and other inpainting methods, with public release planned for dataset, models, and code.

Abstract

Generative image inpainting can produce realistic, high-fidelity results even with large, irregular masks. However, existing methods still face key issues that make inpainted images look unnatural. In this paper, we identify two main problems: (1) Unwanted object insertion: generative models may hallucinate arbitrary objects in the masked region that do not match the surrounding context. (2) Color inconsistency: inpainted regions often exhibit noticeable color shifts, leading to smeared textures and degraded image quality. We analyze the underlying causes of these issues and propose efficient post-hoc solutions for pre-trained inpainting models. Specifically, we introduce the principled framework of Aligned Stable inpainting with UnKnown Areas prior (ASUKA). To reduce unwanted object insertion, we use reconstruction-based priors to guide the generative model, suppressing hallucinated objects while preserving generative flexibility. To address color inconsistency, we design a specialized VAE decoder that formulates latent-to-image decoding as a local harmonization task. This design significantly reduces color shifts and produces more color-consistent results. We implement ASUKA on two representative inpainting architectures: a U-Net-based model and a DiT-based model. We analyze and propose lightweight injection strategies that minimize interference with the model's original generation capacity while ensuring the mitigation of the two issues. We evaluate ASUKA using the Places2 dataset and MISATO, our proposed diverse benchmark. Experiments show that ASUKA effectively suppresses object hallucination and improves color consistency, outperforming standard diffusion, rectified flow models, and other inpainting methods. Dataset, models and codes will be released in github.

Figures (15)

• Figure 1: Image inpainting results obtained using standard SD and FLUX inpainting models, as well as our proposed ASUKA models. ASUKA addresses two fundamental limitations of existing latent inpainting methods: (1) Unwanted object insertion, where spurious elements inconsistent with the unmasked context are generated; (2) Color inconsistency, characterized by noticeable color shifts in the inpainted region that lead to smear-like artifacts. ASUKA introduces a post-training procedure to address these two issues. ASUKA-II suppresses unwanted object insertion more effectively and further improves color consistency compared with ASUKA-I.
• Figure 2: The color shift exists in all kinds of scenarios in inpainted images, including indoor and outdoor scenes, random or continuous masks, and may cause darker or lighter color shift.
• Figure 3: (a) The color of the reconstructed image is shifted, where larger shift is observed during repeated reconstruction. (b) VAE suffers from non-ignorable shifts in low-frequency fields.
• Figure 4: Inpainting w/ v.s. w/o latent augmentation. Inpainting w/o latent augmentation only captures the information loss of VAE, thus still suffer from color inconsistency in some cases. The latent augmentation handles the gap between generated and real latent, further improve the color consistency.
• Figure 5: ASUKA tackles the unwanted object insertion issue by adopting the MAE to provide a stable prior for latent generative models to maintain the generation capacity while mitigating object hallucination. For the color-inconsistency issue, ASUKA utilizes an inpainting-specialized decoder to achieve mask-unmask color consistency when decoding latent.