Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Disentangling to Re-couple: Resolving the Similarity-Controllability Paradox in Subject-Driven Text-to-Image Generation

Shuang Li, Chao Deng, Hang Chen, Liqun Liu, Zhenyu Hu, Te Cao, Mengge Xue, Yuan Chen, Peng Shu, Huan Yu, Jie Jiang

Abstract

Subject-Driven Text-to-Image (T2I) Generation aims to preserve a subject's identity while editing its context based on a text prompt. A core challenge in this task is the "similarity-controllability paradox", where enhancing textual control often degrades the subject's fidelity, and vice-versa. We argue this paradox stems from the ambiguous role of text prompts, which are often tasked with describing both the subject and the desired modifications, leading to conflicting signals for the model. To resolve this, we propose DisCo, a novel framework that first Disntangles and then re-Couples visual and textual information. First, our textual-visual decoupling module isolates the sources of information: subject identity is extracted exclusively from the reference image with the entity word of the subject, while the text prompt is simplified to contain only the modification command, where the subject refers to general pronouns, eliminating descriptive ambiguity. However, this strict separation can lead to unnatural compositions between the subject and its contexts. We address this by designing a dedicated reward signal and using reinforcement learning to seamlessly recouple the visually-defined subject and the textually-generated context. Our approach effectively resolves the paradox, enabling simultaneous high-fidelity subject preservation and precise textual control. Extensive experiments demonstrate that our method achieves state-of-the-art performance, producing highly realistic and coherent images.

Disentangling to Re-couple: Resolving the Similarity-Controllability Paradox in Subject-Driven Text-to-Image Generation

Abstract

Subject-Driven Text-to-Image (T2I) Generation aims to preserve a subject's identity while editing its context based on a text prompt. A core challenge in this task is the "similarity-controllability paradox", where enhancing textual control often degrades the subject's fidelity, and vice-versa. We argue this paradox stems from the ambiguous role of text prompts, which are often tasked with describing both the subject and the desired modifications, leading to conflicting signals for the model. To resolve this, we propose DisCo, a novel framework that first Disntangles and then re-Couples visual and textual information. First, our textual-visual decoupling module isolates the sources of information: subject identity is extracted exclusively from the reference image with the entity word of the subject, while the text prompt is simplified to contain only the modification command, where the subject refers to general pronouns, eliminating descriptive ambiguity. However, this strict separation can lead to unnatural compositions between the subject and its contexts. We address this by designing a dedicated reward signal and using reinforcement learning to seamlessly recouple the visually-defined subject and the textually-generated context. Our approach effectively resolves the paradox, enabling simultaneous high-fidelity subject preservation and precise textual control. Extensive experiments demonstrate that our method achieves state-of-the-art performance, producing highly realistic and coherent images.

Paper Structure

This paper contains 26 sections, 8 equations, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Subject-driven Text-to-image Generation
  4. Reinforcement Learning with Diffusion Models
  5. Method
  6. Preliminary
  7. Textual-visual Decoupling Module
  8. Prompt Simplification.
  9. Visual Grounding.
  10. Analysis of Attention Maps.
  11. Textual-Visual Re-Coupling
  12. Experiments
  13. Experimental Setups
  14. Implementation Details.
  15. Baselines.
  16. ...and 11 more sections

Figures (11)

  • Figure 1: The results with different prompts of the same reference image generated by FLUX Kontext [dev].
  • Figure 2: Visualization of the attention maps between entity word and subject to the generated image, respectively.
  • Figure 3: The architecture of our proposed DisCo framework. DisCo first decouples subject identity (from the image) from textual control (from the prompt). We generate corrupted images to construct preference pairs with the ground truth images to train the Reward Model. Subsequently, it employs GRPO to re-couple textual and visual features, generating a coherent image that preserves subject details while adhering to the prompt.
  • Figure 4: Qualitative comparison with baseline methods on DreamBench.
  • Figure 5: Qualitative results on complex scenarios.
  • ...and 6 more figures