ComposeMe: Attribute-Specific Image Prompts for Controllable Human Image Generation

TL;DR

ComposeMe tackles fine-grained controllability in human image generation by decoupling identity, hairstyle, and clothing into attribute-specific prompts. It introduces dedicated tokenization, simple token merging, and adapter-based injection into a frozen diffusion model, augmented by a two-stage Multi-Attribute Cross-Reference Training regime to learn faithful composition from misaligned attribute inputs. Across multi-attribute multi-ID, single-ID, and full-attribute scenarios, it achieves state-of-the-art prompts-following and attribute fidelity, with ablations validating the critical roles of cross-reference training and attribute-specific tokenization. This work enables modular, configurable human image synthesis with broad potential for personalized generation pipelines and interactive design tools.

Abstract

Generating high-fidelity images of humans with fine-grained control over attributes such as hairstyle and clothing remains a core challenge in personalized text-to-image synthesis. While prior methods emphasize identity preservation from a reference image, they lack modularity and fail to provide disentangled control over specific visual attributes. We introduce a new paradigm for attribute-specific image prompting, in which distinct sets of reference images are used to guide the generation of individual aspects of human appearance, such as hair, clothing, and identity. Our method encodes these inputs into attribute-specific tokens, which are injected into a pre-trained text-to-image diffusion model. This enables compositional and disentangled control over multiple visual factors, even across multiple people within a single image. To promote natural composition and robust disentanglement, we curate a cross-reference training dataset featuring subjects in diverse poses and expressions, and propose a multi-attribute cross-reference training strategy that encourages the model to generate faithful outputs from misaligned attribute inputs while adhering to both identity and textual conditioning. Extensive experiments show that our method achieves state-of-the-art performance in accurately following both visual and textual prompts. Our framework paves the way for more configurable human image synthesis by combining visual prompting with text-driven generation. Webpage is available at: https://snap-research.github.io/composeme/.

Figures (10)

• Figure 1: Conventional single-reference adapter training (left) treats an identity with multiple attributes as a single, indivisible object, cropped directly from the target. In contrast, our Multi-Attribute Cross-Reference Training (right) is attribute-aware: it decomposes identity into distinct visual attributes (face, hair, clothing), sourcing each from different inputs and predicting separate image as target. Our approach curates pairs of misaligned inputs and natural target, therefore enabling generation of coherent outputs even from misaligned attribute inputs in inference.
• Figure 2: Inference pipeline of ComposeMe. Attribute-Specific Image Prompts are processed by specific tokenizers. The merged tokens are injected into a pre-trained diffusion model.
• Figure 3: Qualitative Comparisons in Multi-Attribute, Single-ID Personalization. Compared to OmniGen and GPT-4o, ComposeMe demonstrates significantly improved preservation of ID, hairstyle, and clothing.
• Figure 4: Qualitative Comparisons in Single-Attribute, Single-ID Personalization. We compare ComposeMe with recent PuLID, InfiniteYou, and Omni-ID, all built upon the Flux dev backbone. Ours demonstrates superior expressiveness, accurately following the head pose and expression specified in the text prompt while faithfully preserving subject identity.
• Figure 5: Qualitative Comparisons in Single-ID Full-Body Personalization. Compared to the state-of-the-art methods, our ComposeMe again achieves superior preservation of ID, hair, and clothing attributes.