RealCustom: Narrowing Real Text Word for Real-Time Open-Domain Text-to-Image Customization

TL;DR

RealCustom addresses the entrenched dual-optimum paradox in open-domain text-to-image customization by disentangling subject similarity from text controllability. It introduces a train-inference decoupled framework that, during training, learns generalized visual-text alignment via an adaptive scoring module, and during inference, progressively narrows a real word through adaptive mask guidance to align with the target subject. The method employs a two-branch inference (T2I and TI2I) and integrates a visual cross-attention pathway to regulate influence scope and quantity, enabling real-time, open-domain customization without test-time optimization. Experiments show RealCustom achieves state-of-the-art similarity (CLIP-I, DINO-I) and controllability (CLIP-T, ImageReward) while maintaining efficiency, with robust ablations underscoring the importance of the adaptive scoring and masking strategies.

Abstract

Text-to-image customization, which aims to synthesize text-driven images for the given subjects, has recently revolutionized content creation. Existing works follow the pseudo-word paradigm, i.e., represent the given subjects as pseudo-words and then compose them with the given text. However, the inherent entangled influence scope of pseudo-words with the given text results in a dual-optimum paradox, i.e., the similarity of the given subjects and the controllability of the given text could not be optimal simultaneously. We present RealCustom that, for the first time, disentangles similarity from controllability by precisely limiting subject influence to relevant parts only, achieved by gradually narrowing real text word from its general connotation to the specific subject and using its cross-attention to distinguish relevance. Specifically, RealCustom introduces a novel "train-inference" decoupled framework: (1) during training, RealCustom learns general alignment between visual conditions to original textual conditions by a novel adaptive scoring module to adaptively modulate influence quantity; (2) during inference, a novel adaptive mask guidance strategy is proposed to iteratively update the influence scope and influence quantity of the given subjects to gradually narrow the generation of the real text word. Comprehensive experiments demonstrate the superior real-time customization ability of RealCustom in the open domain, achieving both unprecedented similarity of the given subjects and controllability of the given text for the first time. The project page is https://corleone-huang.github.io/realcustom/.

Figures (11)

• Figure 1: Comparison between the existing paradigm and ours. (a) The existing paradigm represents the given subject as pseudo-words (e.g., $S^*$), which has entangled the same entire influence scope with the given text, resulting in the dual-optimum paradox, i.e., the similarity for the given subject and the controllability for the given text could not achieve optimum simultaneously. (b) We propose RealCustom, a novel paradigm that, for the first time disentangles similarity from controllability by precisely limiting the given subjects to influence only the relevant parts while the rest parts are purely controlled by the given text. This is achieved by iteratively updating the influence scope and influence quantity of the given subjects. (c) The quantitative comparison shows that our paradigm achieves both superior similarity and controllability than the state-of-the-arts of the existing paradigm. CLIP-image score (CLIP-I) and CLIP-text score (CLIP-T) are used to evaluate similarity and controllability. Refer to the experiments for details.
• Figure 2: Generated customization results of our proposed novel paradigm RealCustom. Given a single image representing the given subject in the open domain (any subjects, portrait painting, favorite toys, etc.), RealCustom could generate realistic images that consistently adhere to the given text for the given subjects in real-time (without any test-time optimization steps).
• Figure 3: Illustration of our proposed RealCustom, which employs a novel "train-inference" decoupled framework: (a) During training, general alignment between visual and original text conditions is learned by the proposed adaptive scoring module, which accurately derives visual conditions based on text and currently generated features. (b) During inference, progressively narrowing down a real word (e.g., "toy") from its initial general connotation to the given subject (e.g., the unique brown sloth toy) by the proposed adaptive mask guidance strategy, which consists of two branches, i.e., a text-to-image (T2I) branch where the visual condition is set to $\boldsymbol{0}$, and a text$\&$image-to-image (TI2I) branch where the visual condition is set to the given subject. The T2I branch aims to calculate the influence scope by aggregating the target real word's (e.g., "toy") cross-attention, while the TI2I branch aims to inject the influence quantity into this scope.
• Figure 4: Illustration of adaptive scoring module. Text features and currently generated features are first aggregated into the textual and visual context, which are then spatially concatenated with image features to predict textual and visual scores. These scores are then fused based on the current timestep. Ultimately, only a subset of the key features is selected based on the fused score.
• Figure 5: Qualitative comparison with existing methods. RealCustom could produce much higher quality customization results that have better similarity with the given subject and better controllability with the given text compared to existing works. Moreover, RealCustom shows superior diversity (different subject poses, locations, etc.) and generation quality (e.g., the "autumn leaves" scene in the third row).