CharacterFactory: Sampling Consistent Characters with GANs for Diffusion Models

TL;DR

CharacterFactory presents an end-to-end framework for sampling new identity-consistent characters within diffusion-model pipelines by learning an Identity-Embedding GAN (IDE-GAN) that maps latent codes to CLIP-based celebrity embeddings. A context-consistent loss enforces contextual stability of the generated embeddings, enabling seamless integration with diffusion models without re-training the diffusion backbone. The method achieves fast training (about 10 minutes) and fast inference (new identities in ~3 seconds), outperforming two-stage workflows in identity consistency, editability, and image quality while offering strong interpolation and cross-domain applicability to image/video/3D tasks. This approach enables scalable, controllable identity generation and dataset creation, with practical impact for virtual humans, storytelling, and media production, while maintaining compatibility with existing plug-ins like ControlNet, ModelScopeT2V, and LucidDreamer.

Abstract

Recent advances in text-to-image models have opened new frontiers in human-centric generation. However, these models cannot be directly employed to generate images with consistent newly coined identities. In this work, we propose CharacterFactory, a framework that allows sampling new characters with consistent identities in the latent space of GANs for diffusion models. More specifically, we consider the word embeddings of celeb names as ground truths for the identity-consistent generation task and train a GAN model to learn the mapping from a latent space to the celeb embedding space. In addition, we design a context-consistent loss to ensure that the generated identity embeddings can produce identity-consistent images in various contexts. Remarkably, the whole model only takes 10 minutes for training, and can sample infinite characters end-to-end during inference. Extensive experiments demonstrate excellent performance of the proposed CharacterFactory on character creation in terms of identity consistency and editability. Furthermore, the generated characters can be seamlessly combined with the off-the-shelf image/video/3D diffusion models. We believe that the proposed CharacterFactory is an important step for identity-consistent character generation. Project page is available at: https://qinghew.github.io/CharacterFactory/.

Figures (9)

• Figure 1: Overview of the proposed CharacterFactory. (a) We take the word embeddings of celeb names as ground truths for identity-consistent generation and train a GAN model constructed by MLPs to learn the mapping from $z$ to celeb embedding space. In addition, a context-consistent loss is designed to ensure that the generated pseudo identity can exhibit consistency in various contexts. $s_1^*$, $s_2^*$ are placeholders for $v_1^*$, $v_2^*$. (b) Without diffusion models involved in training, IDE-GAN can end-to-end generate embeddings that can be seamlessly inserted into diffusion models to achieve identity-consistent generation.
• Figure 2: Effect of $\mathcal{L}_{adv}$ and $\mathcal{L}_{con}$. The images in each column are generated by a randomly sampled $z$ and two prompts according to the pipeline in Figure \ref{['fig:framework']}(b). The placeholders $s_1^*$, $s_2^*$ of prompts such as "$s_1^*$$s_2^*$ is smiling" are omitted in this work for brevity (Zoom in for the best view).
• Figure 3: Qualitative comparisons with two-stage workflows using five baselines (denoted with $\dagger$) for creating consistent characters. The upper left corner of the two-stage baselines is the generated image by Stable Diffusion as the input of the second stage. Two-stage workflows struggle to maintain the identity of the generated image and degrade the image quality. In comparison, the proposed CharacterFactory can generate high-quality identity-consistent character images with diverse layouts while conforming to the given text prompts (Zoom in for the best view).
• Figure 4: Qualitative comparisons with the generation results in the papers of two most related methods The Chosen One avrahami2023chosen and ConsiStory tewel2024training. CharacterFactory achieves comparable performance with the same prompts (Zoom in for the best view).
• Figure 5: Interpolation property of IDE-GAN. We conduct linear interpolation between randomly sampled $z_1$ and $z_2$, and generate pseudo identity embeddings with IDE-GAN. To visualize the smooth variations in image space, we insert the generated embeddings into Stable Diffusion via the pipeline of Figure \ref{['fig:framework']}(b). The experiments in row 1, 3 are conducted with the same seeds, and row 2, 4 use random seeds (Zoom in for the best view).