Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeCorStory: Gram-Schmidt Prompt Embedding Decorrelation for Consistent Storytelling

Ayushman Sarkar, Zhenyu Yu, Mohd Yamani Idna Idris

TL;DR

DeCorStory addresses inter-frame semantic interference in training-free multi-frame text-to-image storytelling by explicitly decorrelating frame-level prompt embeddings with Gram-Schmidt, while amplifying frame-specific semantics via Singular-Value Reweighting and stabilizing identity through Identity-Preserving Cross-Attention. The framework operates entirely at inference time, requiring no model fine-tuning and integrating into standard diffusion pipelines. On ConsiStory+ data, it achieves state-of-the-art performance among training-free baselines in prompt alignment and identity consistency, supported by a human user study. Overall, the approach demonstrates that embedding decorrelation coupled with targeted conditioning significantly improves narrative coherence and subject fidelity in diffusion-based storytelling.

Abstract

Maintaining visual and semantic consistency across frames is a key challenge in text-to-image storytelling. Existing training-free methods, such as One-Prompt-One-Story, concatenate all prompts into a single sequence, which often induces strong embedding correlation and leads to color leakage, background blending, and identity drift. We propose DeCorStory, a training-free inference-time framework that explicitly reduces inter-frame semantic interference. DeCorStory applies Gram-Schmidt prompt embedding decorrelation to orthogonalize frame-level semantics, followed by singular value reweighting to strengthen prompt-specific information and identity-preserving cross-attention to stabilize character identity during diffusion. The method requires no model modification or fine-tuning and can be seamlessly integrated into existing diffusion pipelines. Experiments demonstrate consistent improvements in prompt-image alignment, identity consistency, and visual diversity, achieving state-of-the-art performance among training-free baselines. Code is available at: https://github.com/YuZhenyuLindy/DeCorStory

DeCorStory: Gram-Schmidt Prompt Embedding Decorrelation for Consistent Storytelling

TL;DR

DeCorStory addresses inter-frame semantic interference in training-free multi-frame text-to-image storytelling by explicitly decorrelating frame-level prompt embeddings with Gram-Schmidt, while amplifying frame-specific semantics via Singular-Value Reweighting and stabilizing identity through Identity-Preserving Cross-Attention. The framework operates entirely at inference time, requiring no model fine-tuning and integrating into standard diffusion pipelines. On ConsiStory+ data, it achieves state-of-the-art performance among training-free baselines in prompt alignment and identity consistency, supported by a human user study. Overall, the approach demonstrates that embedding decorrelation coupled with targeted conditioning significantly improves narrative coherence and subject fidelity in diffusion-based storytelling.

Abstract

Maintaining visual and semantic consistency across frames is a key challenge in text-to-image storytelling. Existing training-free methods, such as One-Prompt-One-Story, concatenate all prompts into a single sequence, which often induces strong embedding correlation and leads to color leakage, background blending, and identity drift. We propose DeCorStory, a training-free inference-time framework that explicitly reduces inter-frame semantic interference. DeCorStory applies Gram-Schmidt prompt embedding decorrelation to orthogonalize frame-level semantics, followed by singular value reweighting to strengthen prompt-specific information and identity-preserving cross-attention to stabilize character identity during diffusion. The method requires no model modification or fine-tuning and can be seamlessly integrated into existing diffusion pipelines. Experiments demonstrate consistent improvements in prompt-image alignment, identity consistency, and visual diversity, achieving state-of-the-art performance among training-free baselines. Code is available at: https://github.com/YuZhenyuLindy/DeCorStory
Paper Structure (18 sections, 14 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 18 sections, 14 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Consistent text-to-image generation
  4. Training-free consistent storytelling
  5. Prompt and embedding manipulation
  6. Method
  7. Prompt Concatenation
  8. Gram--Schmidt Embedding Decorrelation
  9. Integration with SVR and IPCA
  10. Singular-Value Reweighting (SVR).
  11. Identity-Preserving Cross-Attention (IPCA).
  12. Experiment
  13. Implementation Details
  14. Comparison
  15. User Study
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Overview of the DeCorStory pipeline. The method consists of prompt concatenation, Gram–Schmidt embedding decorrelation, and inference-time modules (SVR and IPCA) for identity-consistent multi-frame generation. The identity prompt is shared across all frames, while decorrelated and reweighted frame-level embeddings provide clean, frame-specific conditioning.
  • Figure 2: Qualitative Comparison. Visual comparison of different T2I generation methods on representative prompts. DeCorStory produces more consistent global–local details, clearer character identity preservation, and better narrative coherence across scenes. More examples see Appendix Figure \ref{['fig:supp1']} and \ref{['fig:supp2']}.
  • Figure A1: Qualitative comparison results.
  • Figure A2: Qualitative comparison results.