Ultra High-Resolution Image Inpainting with Patch-Based Content Consistency Adapter

TL;DR

This paper tackles high-resolution text-guided image inpainting by introducing Patch-Adapter, a dual-stage, lightweight adaptation framework that scales pre-trained 1K diffusion priors to 4K+ without full retraining. Stage 1 employs a Dual Context Adapter to enforce global semantic coherence between masked and unmasked regions, while Stage 2 uses a Reference Patch Adapter with cross-patch attention and CLIP-guided reference patch selection for native-resolution refinement. A hierarchical text-prompting scheme and ControlNet-guided refinement further enhance global coherence and local texture fidelity, achieving state-of-the-art results on OpenImages and the Photo Concept Bucket dataset. The approach demonstrates that patch-wise processing with cross-patch context can sustain prompt adherence and content consistency at ultra-high resolutions, offering a scalable, resource-efficient path for 4K+ inpainting.

Abstract

In this work, we present Patch-Adapter, an effective framework for high-resolution text-guided image inpainting. Unlike existing methods limited to lower resolutions, our approach achieves 4K+ resolution while maintaining precise content consistency and prompt alignment, two critical challenges in image inpainting that intensify with increasing resolution and texture complexity. Patch-Adapter leverages a two-stage adapter architecture to scale the diffusion model's resolution from 1K to 4K+ without requiring structural overhauls: (1) Dual Context Adapter learns coherence between masked and unmasked regions at reduced resolutions to establish global structural consistency; and (2) Reference Patch Adapter implements a patch-level attention mechanism for full-resolution inpainting, preserving local detail fidelity through adaptive feature fusion. This dual-stage architecture uniquely addresses the scalability gap in high-resolution inpainting by decoupling global semantics from localized refinement. Experiments demonstrate that Patch-Adapter not only resolves artifacts common in large-scale inpainting but also achieves state-of-the-art performance on the OpenImages and Photo-Concept-Bucket datasets, outperforming existing methods in both perceptual quality and text-prompt adherence.

Figures (7)

• Figure 1: The proposed Patch-Adapter, which enables text-guided high-resolution inpainting at 4K+ resolution while ensuring global content coherence and producing seamlessly blended, visually harmonious results with high fidelity and rich details.
• Figure 2: As image resolution increases, inpainting models tend to produce more artifacts, undermining overall image quality. (The presentation results are from SDXL-inpainting podell2023sdxl.)
• Figure 3: We propose a two-stage pipeline for high-resolution inpainting. Stage 1 leverages a fine-tuned Dual Context Adapter (DCA) to generate visually coherent and contextually accurate content at a lower resolution while balancing both image context and global text prompts. Stage 2 refines the output by utilizing its upsampled result to preserve global structure and employs Reference Patch Adapter (RPA) to capture cross-patch features, enhancing detail richness and fidelity.
• Figure 4: A qualitative evaluation comparing our proposed pipeline with existing methods. Our approach achieved state‐of‐the‐art performance in content accuracy, visual aesthetics, texture consistency both inside and outside the mask.Furthermore, unlike other models that inadvertently introduce image degradation and blur, our approach generate exceptionally realistic, meticulously defined details in high‐resolution images.
• Figure 5: In our qualitative comparison, the model incorporating Dual Context Adapter (DCA) demonstrates superior performance in challenging scenarios, effectively handling tasks such as object removal, image restoration, and object insertion. In contrast, other methods often struggle with contextual understanding, leading to unpredictable color discrepancies and content artifacts.