One-Prompt-One-Story: Free-Lunch Consistent Text-to-Image Generation Using a Single Prompt

TL;DR

This work addresses maintaining subject identity across frames in text-to-image storytelling without model finetuning. It introduces One-Prompt-One-Story (1Prompt1Story), which concatenates identity and frame prompts into a single input (Prompt Consolidation) and augments generation with Naive Prompt Reweighting, Singular-Value Reweighting (SVR), and Identity-Preserving Cross-Attention (IPCA) to preserve identity and improve frame-level alignment. Evaluations on the ConsiStory+ benchmark show that 1Prompt1Story outperforms training-free baselines and matches or nears training-based methods in prompt alignment while surpassing them in identity consistency, aided by SVR and IPCA. The approach is compatible with multiple diffusion models, supports long stories via a sliding window, and can integrate with control-based methods to extend practical storytelling and animation applications.

Abstract

Text-to-image generation models can create high-quality images from input prompts. However, they struggle to support the consistent generation of identity-preserving requirements for storytelling. Existing approaches to this problem typically require extensive training in large datasets or additional modifications to the original model architectures. This limits their applicability across different domains and diverse diffusion model configurations. In this paper, we first observe the inherent capability of language models, coined context consistency, to comprehend identity through context with a single prompt. Drawing inspiration from the inherent context consistency, we propose a novel training-free method for consistent text-to-image (T2I) generation, termed "One-Prompt-One-Story" (1Prompt1Story). Our approach 1Prompt1Story concatenates all prompts into a single input for T2I diffusion models, initially preserving character identities. We then refine the generation process using two novel techniques: Singular-Value Reweighting and Identity-Preserving Cross-Attention, ensuring better alignment with the input description for each frame. In our experiments, we compare our method against various existing consistent T2I generation approaches to demonstrate its effectiveness through quantitative metrics and qualitative assessments. Code is available at https://github.com/byliutao/1Prompt1Story.

Figures (22)

• Figure 1: Existing methods (up) encounter challenges in consistent T2I generation. T2I models such as SDXL podell2023sdxl and Juggernaut-X-v10 juggernaut_x often exhibit noticeable identity inconsistency across generated images. Although recent methods including IP-Adapter and ConsiStory have improved identity consistency, they lost the alignment between the generated images and corresponding input prompts. Additional results of our 1Prompt1Story (down) demonstrate superior consistency without compromising the alignment between text and images.
• Figure 2: t-SNE visualization of text embeddings (Left):$\boldsymbol{c}^{\mathcal{P}_i}$ from single-prompt generation are closer together compared to those from multi-prompt generation. Statistical results (Right): We evaluated the average distances between the corresponding point sets of all prompt sets on the ConsiStory+ benchmark after dimensionality reduction. The average distance between text embeddings from single-prompt generation is smaller than that from multi-prompt generation.
• Figure 3: (Left): SDXL generates frame images using multi-prompt generation, while Naive Prompt Reweighting (NPR) and our method utilize the single-prompt setup. (Mid): Image features are extracted by DINO-v2 oquab2023dinov2 and visualized by the t-SNE reduction. Naive Prompt Reweighting and 1Prompt1Story show more consistent identity generations than the SDXL model. (Right): Statistics of the average feature distances among generated images from the prompts in our extended ConsiStory+ benchmark, which further confirms that 1Prompt1Story produces better identity consistency.
• Figure 4: (a): The overall pipeline of 1Prompt1Story. We combine the identity prompt and frame prompts into a single prompt, then we apply both Singular-Value Reweighting (SVR) and Identity-Preserving Cross-Attention (IPCA) to generate identity-consistent images. (b): During SVR, we first enhance the semantic information of the express set$\mathcal{X}^{exp}$ (red arrow), then iteratively weaken the semantics for the suppress set$\mathcal{X}^{sup}$ (blue arrow). (c): In IPCA, we concatenate $\tilde{\mathcal{K}}$ with $\bar{\mathcal{K}}$ and $\tilde{\mathcal{V}}$ with $\bar{\mathcal{V}}$ to improve identity consistency.
• Figure 5: Qualitative results. We compare our method with PhotoMaker, IP-Adapter, ConsiStory, and StoryDiffsion. Among them, Texture Inversion, PhotoMaker, ConsiStory, and StoryDiffsion struggled to maintain identity consistency for the dragon object while IP-Adapter produced images with relatively similar poses and backgrounds. See Comparison with the remaining methods in Fig. \ref{['fig: base_all']} of the Appendix.