PixelHacker: Image Inpainting with Structural and Semantic Consistency

TL;DR

PixelHacker introduces Latent Categories Guidance (LCG), a simple yet effective paradigm for diffusion-based image inpainting that injects latent foreground and background features via two fixed-size embeddings into the denoising process with linear attention. Trained on a massive 14-million image-mask dataset annotated with 116 foreground and 21 background categories, PixelHacker achieves strong structural and semantic consistency by guiding generation through foreground semantics, background semantics, and contextual structure without explicit category prompts. Finetuned on Places2, CelebA-HQ, and FFHQ, it delivers state-of-the-art results across FID and LPIPS metrics, showing robust performance on both natural scenes and facial imagery and maintaining coherence under challenging masks. The approach demonstrates significant practical impact for high-quality, semantically coherent inpainting in diverse real-world scenarios, with potential for further category expansion and qualitative improvements evidenced by ablations and qualitative results.

Abstract

Image inpainting is a fundamental research area between image editing and image generation. Recent state-of-the-art (SOTA) methods have explored novel attention mechanisms, lightweight architectures, and context-aware modeling, demonstrating impressive performance. However, they often struggle with complex structure (e.g., texture, shape, spatial relations) and semantics (e.g., color consistency, object restoration, and logical correctness), leading to artifacts and inappropriate generation. To address this challenge, we design a simple yet effective inpainting paradigm called latent categories guidance, and further propose a diffusion-based model named PixelHacker. Specifically, we first construct a large dataset containing 14 million image-mask pairs by annotating foreground and background (potential 116 and 21 categories, respectively). Then, we encode potential foreground and background representations separately through two fixed-size embeddings, and intermittently inject these features into the denoising process via linear attention. Finally, by pre-training on our dataset and fine-tuning on open-source benchmarks, we obtain PixelHacker. Extensive experiments show that PixelHacker comprehensively outperforms the SOTA on a wide range of datasets (Places2, CelebA-HQ, and FFHQ) and exhibits remarkable consistency in both structure and semantics. Project page at https://hustvl.github.io/PixelHacker.

Figures (10)

• Figure 1: Qualitative comparison of our PixelHacker vs. other state-of-the-art methods. While other methods struggle with complex structure and semantics, PixelHacker demonstrates remarkable semantic consistency and superior preservation of texture and color, as shown in Example 1. Moreover, PixelHacker maintains contextual structural consistency and logical coherence, as shown in Example 2.
• Figure 2: Overall pipeline of our PixelHacker. PixelHacker builds upon the latent diffusion architecture by introducing two fixed-size LCG embeddings to separately encode latent foreground and background features. We employ linear attention to inject these latent features into the denoising process, enabling intermittent structural and semantic multiple interactions. This design encourages the model to learn a data distribution that is both structurally and semantically consistent. We elaborate on the interaction details in Fig. \ref{['fig:interaction']} and Sec. \ref{['sec:interaction']}.
• Figure 3: Illustration of various masks we use to construct Latent Categories Guidance (LCG). We assign object semantic masks to the foreground embedding and the other three masks to the background embedding. Details refer to Sec. \ref{['sec:CLCG']}.
• Figure 4: The single interaction process between LCG embeddings and latent features. We elaborate on the details in Sec. \ref{['sec:interaction']}. Throughout the pipeline, multiple interactions are performed in a sequential manner, guiding the model to learn foreground semantics, background semantics, and contextual structures.
• Figure 5: Qualitative comparison of PixelHacker with SOTA methods on Places2. Even under masks that cover almost the entire image, PixelHacker's generated results still exhibit remarkable structural and semantic consistency.