From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generation

TL;DR

Parts2Whole addresses the challenge of controllable human image generation conditioned on multiple appearance parts by introducing a semantic-aware appearance encoder and a shared self-attention mechanism that operate across multiple reference images and the target. A pose encoder and a mask-guided attention module preserve spatial relationships and allow precise selection of appearance parts, enabling high-fidelity portrait customization in a zero-shot, multi-reference setting. Quantitative and qualitative experiments show superior appearance fidelity and alignment with provided references compared to tuning-based and zero-shot baselines, highlighting the method's potential for fine-grained control in applications like virtual try-on and character design. Overall, the framework advances multi-part, reference-conditioned portrait synthesis by maintaining part-level details while ensuring coherent whole-body generation.

Abstract

Recent advancements in controllable human image generation have led to zero-shot generation using structural signals (e.g., pose, depth) or facial appearance. Yet, generating human images conditioned on multiple parts of human appearance remains challenging. Addressing this, we introduce Parts2Whole, a novel framework designed for generating customized portraits from multiple reference images, including pose images and various aspects of human appearance. To achieve this, we first develop a semantic-aware appearance encoder to retain details of different human parts, which processes each image based on its textual label to a series of multi-scale feature maps rather than one image token, preserving the image dimension. Second, our framework supports multi-image conditioned generation through a shared self-attention mechanism that operates across reference and target features during the diffusion process. We enhance the vanilla attention mechanism by incorporating mask information from the reference human images, allowing for the precise selection of any part. Extensive experiments demonstrate the superiority of our approach over existing alternatives, offering advanced capabilities for multi-part controllable human image customization. See our project page at https://huanngzh.github.io/Parts2Whole/.

Figures (10)

• Figure 1: We propose Parts2Whole, which can generate realistic and high-quality human figures in various postures from referential human part images of any quantity and different origins. Our method maintains the high alignment with the corresponding conditional semantic regions, while ensuring diversity and harmony among the whole body.
• Figure 2: Overview of Parts2Whole. Based on the text-to-image diffusion model, our method designs an appearance encoder for encoding various parts of human appearance into multi-scale feature maps. We build this encoder by copying the network structure and pre-trained weights from denoising U-Net. Features obtained from reference images with their textual labels are injected into the generation process by shared attention mechanism layer by layer. To precisely select the specified parts from reference images, we enhance the vanilla self-attention mechanism by incorporating subject masks in the reference images. An illustration of one block in U-Net is shown on the right part.
• Figure 3: Illustration of our Mask-Guided Attention. For each patch $s$ (red point) on the feature map $\bm{F}^{0}$, given subject masks $M^{1:N}$ on the $N$ reference images, we only attend patch $s$ to features in these masks along with the patches on itself.
• Figure 4: Qualitative results generated by Parts2Whole and existing alternatives on our partitioned test set. We do not show the text condition in the figure, but notably, when we input the reference images to our proposed appearance encoder, we will pass in short labels such as face, hair or headwear, upper body clothes, lower body clothes, whole body clothes, shoes, etc.
• Figure 5: Qualitative analysis of using different backbones for the appearance encoder, and our proposed methods.