ComposeAnyone: Controllable Layout-to-Human Generation with Decoupled Multimodal Conditions

TL;DR

This work introduces ComposeAnyone, a diffusion-based framework for controllable human image generation that decouples text, reference images, and hand-drawn layouts to drive synthesis. It enables multi-modal inputs through hand-drawn color-block layouts for precise spatial control, multiple reference images for pixel-level fusion, and text descriptions, all fused via latent-space concatenation and attention modulation. A new ComposeHuman dataset provides decoupled annotations for body parts, text, and references to support diverse tasks. Empirical results on VITON-HD, DressCode, and DeepFashion2 show improved alignment to layout, description, and references, demonstrating strong performance in both layout-guided and subject-driven generation tasks and highlighting practical potential for rapid, customizable fashion image creation.

Abstract

Building on the success of diffusion models, significant advancements have been made in multimodal image generation tasks. Among these, human image generation has emerged as a promising technique, offering the potential to revolutionize the fashion design process. However, existing methods often focus solely on text-to-image or image reference-based human generation, which fails to satisfy the increasingly sophisticated demands. To address the limitations of flexibility and precision in human generation, we introduce ComposeAnyone, a controllable layout-to-human generation method with decoupled multimodal conditions. Specifically, our method allows decoupled control of any part in hand-drawn human layouts using text or reference images, seamlessly integrating them during the generation process. The hand-drawn layout, which utilizes color-blocked geometric shapes such as ellipses and rectangles, can be easily drawn, offering a more flexible and accessible way to define spatial layouts. Additionally, we introduce the ComposeHuman dataset, which provides decoupled text and reference image annotations for different components of each human image, enabling broader applications in human image generation tasks. Extensive experiments on multiple datasets demonstrate that ComposeAnyone generates human images with better alignment to given layouts, text descriptions, and reference images, showcasing its multi-task capability and controllability.

Figures (7)

• Figure 1: ComposeAnyone is capable of generating high-quality human images with decoupled multimodal conditions, such as captions, reference images, and hand-drawn layouts.
• Figure 2: Overview of ComposeAnyone. $(a)$Data Preparation. We leverage CogVLM2, SAM, and SCHP to enrich the image-based try-on datasets with fine-grained textual descriptions, hand-drawn layouts, and human component sets. $(b)$Training Procedure. We begin by employing the VAE encoder and the CLIP encoder to extract image and text embeddings, respectively, subsequently injecting the embeddings into the U-Net through concatenation across space and channel dimensions, yielding impressive results without the necessity of additional feature networks.
• Figure 3: An example of fine-grained text description of $Top$. We use CogVLM2 to extract various attributes corresponding to components. Finally, these attributes are combined and transformed into a single descriptive sentence.
• Figure 4: Qualitative comparison with subject-driven methods. ComposeAnyone demonstrates high fidelity in matching specific features of a given reference cloth image.
• Figure 5: Qualitative comparison with layout-guided text-to-image methods. ComposeAnyone demonstrates a high level of congruity with both textual descriptions and spatial layout arrangements in its generative output.