Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Story-Adapter: A Training-free Iterative Framework for Long Story Visualization

Jiawei Mao, Xiaoke Huang, Yunfei Xie, Yuanqi Chang, Mude Hui, Bingjie Xu, Yuyin Zhou

TL;DR

Story-Adapter presents a training-free iterative framework that refines long-story visualizations by leveraging all previously generated frames. A plug-and-play Global Reference Cross-Attention (GRCA) module aggregates global embeddings via CLIP to maintain semantic coherence across up to 100 frames, while a linear weighting schedule balances text guidance and global consistency. The approach is validated on regular-length and long-story tasks, showing improvements in semantic consistency and fine-grained interactions, with ablations confirming the benefits of GRCA, iterative refinement, and initialization. The work demonstrates that iterative, reference-rich guidance can substantially enhance long-form text-to-image story visualization without model re-training, offering a scalable path for high-quality narrative imagery.

Abstract

Story visualization, the task of generating coherent images based on a narrative, has seen significant advancements with the emergence of text-to-image models, particularly diffusion models. However, maintaining semantic consistency, generating high-quality fine-grained interactions, and ensuring computational feasibility remain challenging, especially in long story visualization (i.e., up to 100 frames). In this work, we propose a training-free and computationally efficient framework, termed Story-Adapter, to enhance the generative capability of long stories. Specifically, we propose an iterative paradigm to refine each generated image, leveraging both the text prompt and all generated images from the previous iteration. Central to our framework is a training-free global reference cross-attention module, which aggregates all generated images from the previous iteration to preserve semantic consistency across the entire story, while minimizing computational costs with global embeddings. This iterative process progressively optimizes image generation by repeatedly incorporating text constraints, resulting in more precise and fine-grained interactions. Extensive experiments validate the superiority of Story-Adapter in improving both semantic consistency and generative capability for fine-grained interactions, particularly in long story scenarios. The project page and associated code can be accessed via https://jwmao1.github.io/storyadapter .

Story-Adapter: A Training-free Iterative Framework for Long Story Visualization

TL;DR

Story-Adapter presents a training-free iterative framework that refines long-story visualizations by leveraging all previously generated frames. A plug-and-play Global Reference Cross-Attention (GRCA) module aggregates global embeddings via CLIP to maintain semantic coherence across up to 100 frames, while a linear weighting schedule balances text guidance and global consistency. The approach is validated on regular-length and long-story tasks, showing improvements in semantic consistency and fine-grained interactions, with ablations confirming the benefits of GRCA, iterative refinement, and initialization. The work demonstrates that iterative, reference-rich guidance can substantially enhance long-form text-to-image story visualization without model re-training, offering a scalable path for high-quality narrative imagery.

Abstract

Story visualization, the task of generating coherent images based on a narrative, has seen significant advancements with the emergence of text-to-image models, particularly diffusion models. However, maintaining semantic consistency, generating high-quality fine-grained interactions, and ensuring computational feasibility remain challenging, especially in long story visualization (i.e., up to 100 frames). In this work, we propose a training-free and computationally efficient framework, termed Story-Adapter, to enhance the generative capability of long stories. Specifically, we propose an iterative paradigm to refine each generated image, leveraging both the text prompt and all generated images from the previous iteration. Central to our framework is a training-free global reference cross-attention module, which aggregates all generated images from the previous iteration to preserve semantic consistency across the entire story, while minimizing computational costs with global embeddings. This iterative process progressively optimizes image generation by repeatedly incorporating text constraints, resulting in more precise and fine-grained interactions. Extensive experiments validate the superiority of Story-Adapter in improving both semantic consistency and generative capability for fine-grained interactions, particularly in long story scenarios. The project page and associated code can be accessed via https://jwmao1.github.io/storyadapter .
Paper Structure (49 sections, 4 equations, 16 figures, 6 tables, 1 algorithm)

This paper contains 49 sections, 4 equations, 16 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Diffusion Models
  4. Story Visualization
  5. Subject-Consistent Image Generation
  6. Method
  7. Initialization
  8. Story-Adapter
  9. Global Reference Cross-Attention
  10. Experiments
  11. Dataset and Evaluation
  12. Implementation Details
  13. Regular-length Story Visualization
  14. Quantitative Evaluation.
  15. Qualitative Evaluation.
  16. ...and 34 more sections

Figures (16)

  • Figure 1: A long story of “snowman" visualized by our Story-Adapter from different iterations, compared with those visualized by previous StoryDiffusion zhou2024storydiffusion and StoryGen liu2024intelligent. Notable differences are highlighted in green and red. Zoom in for a better view.
  • Figure 2: Comparison of paradigms for long story visualization: (A) Auto-Regressive (AR): generates frames sequentially referencing on previous finite frames (e.g. the previous three frames); (B) Reference-Image (RI): employs fixed reference images (e.g. the beginning four frames) as reference images; (C) Iterative Paradigm: leverages all frames from the previous iteration as reference images.
  • Figure 3: Illustration of the proposed iterative paradigm, which consists of initialization, iterations in Story-Adapter, and implementation of Global Reference Cross-Attention.
  • Figure 4: Qualitative comparisons for regular-length story visualization. Zoom in for a better view.
  • Figure 5: Qualitative comparisons for long story visualization. The image sequences in orange and blue boxes are generated by StoryDiffusion and Story-Adapter, respectively. Story-Adapter shows advantages in generating semantic consistency and character interactions. Zoom in for a better view.
  • ...and 11 more figures