UMO: Scaling Multi-Identity Consistency for Image Customization via Matching Reward

TL;DR

UMO tackles the scalability challenge of preserving identity across multiple reference identities in image customization. It reframes multi-identity generation as a global assignment problem and optimizes it through Reference Reward Feedback Learning, introducing a Multi-Identity Matching Reward to bind references to generated faces. A scalable data pipeline and the ID-Conf metric quantify identity confusion, and empirical results on XVerseBench and OmniContext show significant gains in identity similarity with reduced confusion, achieving state-of-the-art performance among open-source methods. Overall, UMO enhances identity fidelity and scalability across single and multi-identity scenarios, offering a practical framework for diffusion-based image customization.

Abstract

Recent advancements in image customization exhibit a wide range of application prospects due to stronger customization capabilities. However, since we humans are more sensitive to faces, a significant challenge remains in preserving consistent identity while avoiding identity confusion with multi-reference images, limiting the identity scalability of customization models. To address this, we present UMO, a Unified Multi-identity Optimization framework, designed to maintain high-fidelity identity preservation and alleviate identity confusion with scalability. With "multi-to-multi matching" paradigm, UMO reformulates multi-identity generation as a global assignment optimization problem and unleashes multi-identity consistency for existing image customization methods generally through reinforcement learning on diffusion models. To facilitate the training of UMO, we develop a scalable customization dataset with multi-reference images, consisting of both synthesised and real parts. Additionally, we propose a new metric to measure identity confusion. Extensive experiments demonstrate that UMO not only improves identity consistency significantly, but also reduces identity confusion on several image customization methods, setting a new state-of-the-art among open-source methods along the dimension of identity preserving. Code and model: https://github.com/bytedance/UMO

Figures (12)

• Figure 1: Showcase of our UMO model in different scenarios. The detailed prompts are listed in \ref{['tab:showcase_prompt']}.
• Figure 2: Our UMO unleashes multi-identity consistency and alleviates identity confusion. Existing image customization methods suffer low facial fidelity and severe identity confusion, while UMO can tackle these problems with results in blue boxes.
• Figure 3: Illustration of the training framework of UMO. UMO's training process follows ReReFL in \ref{['alg:rerefl']} with Multi-Identity Matching Reward.
• Figure 4: Single identity reward (SIR) scores of UNO uno and OmniGen2 omnigen2 with different generation seeds along denoising steps. The scores become stable after step 5 and 10 respectively. And the results with highest and lowest reward scores indicating its discriminatory ability.
• Figure 5: Qualitative comparison with different methods on XVerseBench xverse.