Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WithAnyone: Towards Controllable and ID Consistent Image Generation

Hengyuan Xu, Wei Cheng, Peng Xing, Yixiao Fang, Shuhan Wu, Rui Wang, Xianfang Zeng, Daxin Jiang, Gang Yu, Xingjun Ma, Yu-Gang Jiang

TL;DR

This work tackles copy-paste artifacts in identity-consistent image generation by introducing a large paired MultiID-2M dataset and a standardized MultiID-Bench benchmark for multi-identity synthesis. It then presents WithAnyone, a diffusion-based model built on FLUX that uses GT-aligned ID loss and extended-negative ID contrastive learning to achieve high identity fidelity while significantly reducing copy-paste. A four-phase training pipeline—from reconstruction pre-training to paired tuning and quality/style refinement—enables robust identity conditioning and controllability across poses and expressions. Extensive quantitative, qualitative, and user-study evaluations demonstrate that WithAnyone outperforms baselines in identity preservation and perceptual quality, breaking the traditional fidelity-versus-copying trade-off. These contributions provide a practical pathway toward more expressive and controllable multi-identity image generation in real-world applications.

Abstract

Identity-consistent generation has become an important focus in text-to-image research, with recent models achieving notable success in producing images aligned with a reference identity. Yet, the scarcity of large-scale paired datasets containing multiple images of the same individual forces most approaches to adopt reconstruction-based training. This reliance often leads to a failure mode we term copy-paste, where the model directly replicates the reference face rather than preserving identity across natural variations in pose, expression, or lighting. Such over-similarity undermines controllability and limits the expressive power of generation. To address these limitations, we (1) construct a large-scale paired dataset MultiID-2M, tailored for multi-person scenarios, providing diverse references for each identity; (2) introduce a benchmark that quantifies both copy-paste artifacts and the trade-off between identity fidelity and variation; and (3) propose a novel training paradigm with a contrastive identity loss that leverages paired data to balance fidelity with diversity. These contributions culminate in WithAnyone, a diffusion-based model that effectively mitigates copy-paste while preserving high identity similarity. Extensive qualitative and quantitative experiments demonstrate that WithAnyone significantly reduces copy-paste artifacts, improves controllability over pose and expression, and maintains strong perceptual quality. User studies further validate that our method achieves high identity fidelity while enabling expressive controllable generation.

WithAnyone: Towards Controllable and ID Consistent Image Generation

TL;DR

This work tackles copy-paste artifacts in identity-consistent image generation by introducing a large paired MultiID-2M dataset and a standardized MultiID-Bench benchmark for multi-identity synthesis. It then presents WithAnyone, a diffusion-based model built on FLUX that uses GT-aligned ID loss and extended-negative ID contrastive learning to achieve high identity fidelity while significantly reducing copy-paste. A four-phase training pipeline—from reconstruction pre-training to paired tuning and quality/style refinement—enables robust identity conditioning and controllability across poses and expressions. Extensive quantitative, qualitative, and user-study evaluations demonstrate that WithAnyone outperforms baselines in identity preservation and perceptual quality, breaking the traditional fidelity-versus-copying trade-off. These contributions provide a practical pathway toward more expressive and controllable multi-identity image generation in real-world applications.

Abstract

Identity-consistent generation has become an important focus in text-to-image research, with recent models achieving notable success in producing images aligned with a reference identity. Yet, the scarcity of large-scale paired datasets containing multiple images of the same individual forces most approaches to adopt reconstruction-based training. This reliance often leads to a failure mode we term copy-paste, where the model directly replicates the reference face rather than preserving identity across natural variations in pose, expression, or lighting. Such over-similarity undermines controllability and limits the expressive power of generation. To address these limitations, we (1) construct a large-scale paired dataset MultiID-2M, tailored for multi-person scenarios, providing diverse references for each identity; (2) introduce a benchmark that quantifies both copy-paste artifacts and the trade-off between identity fidelity and variation; and (3) propose a novel training paradigm with a contrastive identity loss that leverages paired data to balance fidelity with diversity. These contributions culminate in WithAnyone, a diffusion-based model that effectively mitigates copy-paste while preserving high identity similarity. Extensive qualitative and quantitative experiments demonstrate that WithAnyone significantly reduces copy-paste artifacts, improves controllability over pose and expression, and maintains strong perceptual quality. User studies further validate that our method achieves high identity fidelity while enabling expressive controllable generation.
Paper Structure (35 sections, 11 equations, 17 figures, 5 tables)

This paper contains 35 sections, 11 equations, 17 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Single-ID Preservation.
  4. Multi-ID Preservation.
  5. ID-Centric Datasets and Benchmarks.
  6. MultiID-2M : Paired Multi-Person Dataset Construction
  7. MultiID-Bench : Comprehensive ID Customization Evaluation
  8. WithAnyone : Controllable and ID-Consistent Generation
  9. Training Objectives
  10. Training pipeline
  11. Experiments
  12. Quantitative Evaluation
  13. Qualitative Comparison
  14. Ablation and User Studies
  15. Conclusion
  16. ...and 20 more sections

Figures (17)

  • Figure 1: Showcases of WithAnyone. WithAnyone is capable of generating high-quality, controllable, and ID-consistent images by leveraging ID-contrastive training on the proposed MultiID-2M dataset.
  • Figure 2: Our Observation. Natural variations, such as head pose, expression, and makeup, may cause more face similarity decrease than expected. Copying reference image limits models' ability to respond to expression and makeup adjustment prompts.
  • Figure 3: Overview of WithAnyone. It builds on a large-scale dataset, MultiID-2M, constructed through a four-step pipeline: (1) collect and cluster single-ID data based on identity similarity; (2) gather multi-ID data via targeted searches using desired identity names with negative keywords for filtering; (3) form image pairs by matching faces between single-ID and multi-ID data; and (4) apply post-processing for quality control and stylization. Training proceeds in four stages: (1) pre-train on single-ID, multi-ID, and open-domain images with fixed prompts; (2) train with image-caption supervision; (3) fine-tune with ID-paired data; and (4) perform quality tuning using a curated high-quality subset.
  • Figure 4: (a) Architecture of WithAnyone: Each reference is encoded by both a face-recognition network and a general image encoder, yielding identity-discriminative signals and complementary mid-level features. Face embeddings are restricted to attend only to image tokens within their corresponding face regions. (b) Training Objectives of WithAnyone: In addition to the diffusion loss, we incorporate an ID contrastive loss and a ground-truth–aligned ID loss, which together provide consistent and accurate identity supervision.
  • Figure 5: Trade-off between Face Similarity and Copy-paste. Except for WithAnyone, the other models fall roughly on a fitted curve, illustrating a clear trade-off between face similarity and copy-paste. Upper-right corner is desired.
  • ...and 12 more figures