Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Consistent text-to-image generation via scene de-contextualization

Song Tang, Peihao Gong, Kunyu Li, Kai Guo, Boyu Wang, Mao Ye, Jianwei Zhang, Xiatian Zhu

TL;DR

This work identifies scene contextualization as a principal source of identity shift in text-to-image generation, arising from attention-based coupling between subject prompts and scene context. It provides theoretical bounds on contextualization strength and introduces Scene De-Contextualization (SDeC), a training-free prompt-embedding editing method that detects and suppresses the latent scene–ID correlation subspace via forward-backward SVD optimization and eigenvalue reweighting, enabling per-scene prompts without requiring knowledge of all target scenes. Empirically, SDeC delivers improved identity preservation while maintaining scene diversity across multiple backbones and tasks, and it demonstrates compatibility with additional generative controls (e.g., pose maps) and complementary methods. The results offer a flexible, scalable approach for real-world applications where target scenes are dynamic or unavailable in advance, advancing practical consistency in T2I systems.

Abstract

Consistent text-to-image (T2I) generation seeks to produce identity-preserving images of the same subject across diverse scenes, yet it often fails due to a phenomenon called identity (ID) shift. Previous methods have tackled this issue, but typically rely on the unrealistic assumption of knowing all target scenes in advance. This paper reveals that a key source of ID shift is the native correlation between subject and scene context, called scene contextualization, which arises naturally as T2I models fit the training distribution of vast natural images. We formally prove the near-universality of this scene-ID correlation and derive theoretical bounds on its strength. On this basis, we propose a novel, efficient, training-free prompt embedding editing approach, called Scene De-Contextualization (SDeC), that imposes an inversion process of T2I's built-in scene contextualization. Specifically, it identifies and suppresses the latent scene-ID correlation within the ID prompt's embedding by quantifying the SVD directional stability to adaptively re-weight the corresponding eigenvalues. Critically, SDeC allows for per-scene use (one scene per prompt) without requiring prior access to all target scenes. This makes it a highly flexible and general solution well-suited to real-world applications where such prior knowledge is often unavailable or varies over time. Experiments demonstrate that SDeC significantly enhances identity preservation while maintaining scene diversity.

Consistent text-to-image generation via scene de-contextualization

TL;DR

This work identifies scene contextualization as a principal source of identity shift in text-to-image generation, arising from attention-based coupling between subject prompts and scene context. It provides theoretical bounds on contextualization strength and introduces Scene De-Contextualization (SDeC), a training-free prompt-embedding editing method that detects and suppresses the latent scene–ID correlation subspace via forward-backward SVD optimization and eigenvalue reweighting, enabling per-scene prompts without requiring knowledge of all target scenes. Empirically, SDeC delivers improved identity preservation while maintaining scene diversity across multiple backbones and tasks, and it demonstrates compatibility with additional generative controls (e.g., pose maps) and complementary methods. The results offer a flexible, scalable approach for real-world applications where target scenes are dynamic or unavailable in advance, advancing practical consistency in T2I systems.

Abstract

Consistent text-to-image (T2I) generation seeks to produce identity-preserving images of the same subject across diverse scenes, yet it often fails due to a phenomenon called identity (ID) shift. Previous methods have tackled this issue, but typically rely on the unrealistic assumption of knowing all target scenes in advance. This paper reveals that a key source of ID shift is the native correlation between subject and scene context, called scene contextualization, which arises naturally as T2I models fit the training distribution of vast natural images. We formally prove the near-universality of this scene-ID correlation and derive theoretical bounds on its strength. On this basis, we propose a novel, efficient, training-free prompt embedding editing approach, called Scene De-Contextualization (SDeC), that imposes an inversion process of T2I's built-in scene contextualization. Specifically, it identifies and suppresses the latent scene-ID correlation within the ID prompt's embedding by quantifying the SVD directional stability to adaptively re-weight the corresponding eigenvalues. Critically, SDeC allows for per-scene use (one scene per prompt) without requiring prior access to all target scenes. This makes it a highly flexible and general solution well-suited to real-world applications where such prior knowledge is often unavailable or varies over time. Experiments demonstrate that SDeC significantly enhances identity preservation while maintaining scene diversity.
Paper Structure (40 sections, 4 theorems, 26 equations, 16 figures, 4 tables)

This paper contains 40 sections, 4 theorems, 26 equations, 16 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Scene Contextualization
  4. Theorem behind Scene Contextualization
  5. Bounding the Strength of Contextualization
  6. Scene De-Contextualization
  7. Overview
  8. Identifying latent scene-ID correlation subspace in $\mathcal{Z}_{\mathrm{id}}^{\rm o}$
  9. De-contextualization by suppressing latent scene-ID correlation subspace in $\mathcal{Z}_{\mathrm{id}}^{\rm o}$
  10. Experiments
  11. Results analysis
  12. User study
  13. Further Analysis
  14. Ablation study
  15. Validation of scene contextualization control
  16. ...and 25 more sections

Key Result

Theorem 1

Let $\mathcal{H}_{\rm{id}},\mathcal{H}_{\rm{sc}}\subset\mathbb R^{d}$ be the identity/scene semantic subspaces with ideal semantic separation: $\mathcal{H}_{\rm{id}} \cap \mathcal{H}_{\rm{sc}} = \varnothing$; $\mathcal{Z}_{\rm{id}}\subseteq \mathcal{H}_{\rm{id}}$ and $\mathcal{Z}_{\rm{sc}}^k\subsete where the attention weights by token index $\alpha=\left[ \alpha_{\mathrm{id}},\alpha_{\mathrm{sc}}

Figures (16)

  • Figure 1: Illustration of scene contextualization with SDXL. Left: The attire of the subject varies with the site. Right: The subject’s clothing changes with the season.
  • Figure 2: Example of scene contextualization with ID prompt "a photo of a dog" and scene prompt "chasing a frisbee in a park". Left: Correlation between tokens and the attention similarity matrix reveals that scene tokens influence ID generation (visualization method follows hertz2023prompt). Right: Similarity between SVD eigenvectors of ID and scene embedding shows they share an overlapping subspace.
  • Figure 3: Overview of SDeC. A text prompt $\mathcal{P}^{k}=\mathcal{P}_{\rm id}\oplus\mathcal{P}_{\rm sc}^{k}$ is encoded into prompt embeddings $[\mathcal{Z}_{\rm id}^{\rm o};\mathcal{Z}_{\rm sc}^{k}]$ where $^{\rm o}$ means "original". SDeC mitigates scene contextualization by (a) identifying and (b) suppressing latent scene-ID correlation in $\mathcal{Z}_{\rm id}^{\rm o}$ (QDV: Quantifying Directional influence Variations). The refined ID embedding $\mathcal{Z}_{\rm id}^{*}$ is then concatenated with $\mathcal{Z}_{\rm sc}^{k}$ for subsequent generation.
  • Figure 4: Qualitative results. CS: ConsiStory.
  • Figure 5: Validation of scene contextualization control. Top: PCA-based analysis where the image pairs blocked by the green-highlighted box have noticeable discrepancy. Bottom: Correlation between scene token embedding and attention similarity matrix.
  • ...and 11 more figures

Theorems & Definitions (8)

  • Theorem 1
  • Corollary 1
  • Theorem 2
  • Corollary 2
  • Proof 1
  • Proof 2
  • Proof 3
  • Proof 4