Visual Persona: Foundation Model for Full-Body Human Customization

TL;DR

Visual Persona tackles full-body text-to-image customization from a single real image by introducing a region-aware transformer architecture that produces dense identity embeddings to condition a frozen diffusion model. It overcomes data scarcity with Visual Persona-500K, a large paired full-body dataset curated via vision-language models and Phi-3 captions to separate identity from intra-individual variation. The method partitions the input into five body regions, encodes local appearance with DINOv2, and uses a body-part transformer to generate dense identity embeddings that guide diffusion with decoupled identity cross-attention alongside text prompts, achieving high-fidelity, identity-consistent outputs across diverse poses and scenes. Evaluations on GPT-based Dreambench++ and human studies show Visual Persona outperforms prior methods in identity preservation and text alignment, while enabling versatile applications such as part-guided generation, multi-person customization, and text-guided virtual try-on; limitations include body-proportion inaccuracies and leakage of identity-unrelated background cues, suggesting avenues for improved foreground masking and anatomy fidelity.

Abstract

We introduce Visual Persona, a foundation model for text-to-image full-body human customization that, given a single in-the-wild human image, generates diverse images of the individual guided by text descriptions. Unlike prior methods that focus solely on preserving facial identity, our approach captures detailed full-body appearance, aligning with text descriptions for body structure and scene variations. Training this model requires large-scale paired human data, consisting of multiple images per individual with consistent full-body identities, which is notoriously difficult to obtain. To address this, we propose a data curation pipeline leveraging vision-language models to evaluate full-body appearance consistency, resulting in Visual Persona-500K, a dataset of 580k paired human images across 100k unique identities. For precise appearance transfer, we introduce a transformer encoder-decoder architecture adapted to a pre-trained text-to-image diffusion model, which augments the input image into distinct body regions, encodes these regions as local appearance features, and projects them into dense identity embeddings independently to condition the diffusion model for synthesizing customized images. Visual Persona consistently surpasses existing approaches, generating high-quality, customized images from in-the-wild inputs. Extensive ablation studies validate design choices, and we demonstrate the versatility of Visual Persona across various downstream tasks.

Figures (33)

• Figure 1: Visual Persona enables Text-to-Image Full-Body Human Customization. Given a single human image, our method generates diverse customized images of the individual that are closely aligned with text descriptions while accurately preserving full-body appearance, which unlocks a wide range of applications, including (a) text-guided virtual try-on, (b) human stylization, and (c) character customization.
• Figure 2: Data Statistics: Our curated training dataset, Visual Persona-500K, consists of 580k images representing 100k individuals. (a) illustrates the distribution of the number of images per individual, with over 50% of individuals having more than four images, and shows example image-caption pairs from the same individual. (b) highlights the diversity of individuals based on facial attributes, including race, age, and gender, which are estimated by DeepFace serengil2021lightface. (c) showcases body structure diversity, segmented into five clusters—full-body, face, torso, legs, and shoes—categorized using a body-parsing method li2020self.
• Figure 3: Overall Architecture: Our network augments the input human image into body regions, which are encoded into local features by an image transformer encoder. A body-partitioned transformer decoder projects these features into learnable identity embeddings via cross-attention, followed by self-attention and MLP. After $M$ iterations, the embeddings are concatenated to form a stacked identity embedding. The identity embedding and text embedding from detailed captions condition a pre-trained T2I diffusion model to synthesize a new image with the input identity. Only the body-partitioned transformer decoder and identity cross-attention module are trained.
• Figure 4: Body Part Decomposition.
• Figure 5: Qualitative Comparison on PPR10K liang2021ppr10k: Compared to prior works that focus on face identity preservation ye2023ipwang2024instantidli2024photomaker or fail to capture the input's detailed appearance ye2023ipzhou2024storymaker, Visual Persona accurately preserves the full-body appearance while generating diverse images based on text prompts.