CompleteMe: Reference-based Human Image Completion

TL;DR

CompleteMe tackles the challenge of preserving unique, person-specific details in human image completion by introducing a dual U-Net architecture ($U_{ref}$ and $U_{comp}$) augmented with Region-focused Attention (RFA) to enforce precise correspondences with reference images. The framework encodes multi-part reference visuals via a Reference U-Net (initialized from Stable Diffusion $1.5$ at timestep $0$) and fuses them into the Complete U-Net through RFA, guided by global CLIP features for semantic coherence. A dedicated benchmark built from the UniHuman/Wpose corpus evaluates the model under significant pose variation and identity-critical details, using metrics like CLIP-I, CLIP-T, DINO, DreamSim, PSNR, SSIM, and LPIPS, complemented by a user study. Experiments show CompleteMe outperforms both non-reference and reference-based baselines in preserving identity and fine details, demonstrating robust performance even with single-reference inputs and optional textual prompts, with potential applications in editing, virtual try-on, and animation.

Abstract

Recent methods for human image completion can reconstruct plausible body shapes but often fail to preserve unique details, such as specific clothing patterns or distinctive accessories, without explicit reference images. Even state-of-the-art reference-based inpainting approaches struggle to accurately capture and integrate fine-grained details from reference images. To address this limitation, we propose CompleteMe, a novel reference-based human image completion framework. CompleteMe employs a dual U-Net architecture combined with a Region-focused Attention (RFA) Block, which explicitly guides the model's attention toward relevant regions in reference images. This approach effectively captures fine details and ensures accurate semantic correspondence, significantly improving the fidelity and consistency of completed images. Additionally, we introduce a challenging benchmark specifically designed for evaluating reference-based human image completion tasks. Extensive experiments demonstrate that our proposed method achieves superior visual quality and semantic consistency compared to existing techniques. Project page: https://liagm.github.io/CompleteMe/

Figures (6)

• Figure 1: Given occluded human image, non-reference methods, LOHC zhao2024large and BrushNet ju2024brushnet, can generate plausible results but lack the unique information of the person like special clothing and tattoo pattern (highlighted in Red box). Such information can be only acquired by additional reference images. Given the reference image, MimicBrush chen2024zero fails to find the corresponding parts between input and reference. Our CompleteMe can preserve identical and fine-detail information from the reference image and generate a consistent result.
• Figure 2: CompleteMe Pipeline. Our proposed CompleteMe utilizes a dual U-Net framework composed of a Reference U-Net ($U_{ref}$) and a Complete U-Net ($U_{comp}$). Given an input image ($I_{input}$) with masked regions, we first encode the input image to latent feature $f_{input}$. The Reference U-Net then extracts detailed visual features ($f_{ref}^{0}, f_{ref}^{1}, \dots, f_{ref}^{n}$) from multiple reference images ($I_{ref}$), which consist of different human body parts. Along with global semantic features ($f_{\text{CLIP}}$) extracted by CLIP, the reference features are processed within our novel Region-focused Attention (RFA) Block embedded in the Complete U-Net. These reference features are then explicitly masked according to reference masks, producing masked reference features (${f'}_{ref}^{0}, {f'}_{ref}^{1}, \dots, {f'}_{ref}^{n}$). This explicit masking and concatenation strategy enables the model to precisely zoom in and focus on relevant human regions, establishing accurate and fine-grained correspondences through the Region-focused Attention mechanism. Finally, decoupled cross-attention integrates these refined local features with the global semantic CLIP features ($f_{\text{CLIP}}$), resulting in a detailed and semantically coherent completion.
• Figure 3: Qualitative Comparison with Non-reference Methods. We compare CompleteMe with non-reference methods, LOHC zhao2024large and BrushNet ju2024brushnet. Given masked inputs, these non-reference methods generate plausible content for the masked regions using image priors or text prompts. However, as indicated in the Red box, they cannot reproduce specific details such as tattoos or unique clothing patterns, as they lack reference images to guide the reconstruction of identical information.
• Figure 4: Qualitative Comparison with Reference-based Methods. Our CompleteMe can generate more realistic and preserve identical information from the reference image. Please refer to the Red box region for a more detailed comparison.
• Figure 5: Qualitative Comparison on Different Reference Image Encoder. We conduct the ablation study for different encoders to extract the feature from reference images. CLIP radford2021learning and DINOv2 oquab2024dinov2 can find the correspondence between masked input and the reference image, but they can not preserve the detailed information compared to the U-Net encoder. For the effectiveness of our Region-focused Attention (RFA), this design further helps preserve the identical information. Please zoom in for the detail inside the Red box.