Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Models

TL;DR

Open-ended visual storytelling aims to generate coherent image sequences from storylines. The authors introduce StoryGen, a latent-diffusion, auto-regressive generator with a novel vision-language context module that conditions on current prompts and prior image–caption pairs, plus StorySalon, a large dataset of diverse, animation-style storybooks. Quantitative and human evaluations show StoryGen achieves improved image quality, visual-language alignment, and character consistency, and generalizes to unseen characters without test-time optimization. This work enables scalable, open-ended visual storytelling across varied topics and styles, with potential educational and creative impacts.

Abstract

Generative models have recently exhibited exceptional capabilities in text-to-image generation, but still struggle to generate image sequences coherently. In this work, we focus on a novel, yet challenging task of generating a coherent image sequence based on a given storyline, denoted as open-ended visual storytelling. We make the following three contributions: (i) to fulfill the task of visual storytelling, we propose a learning-based auto-regressive image generation model, termed as StoryGen, with a novel vision-language context module, that enables to generate the current frame by conditioning on the corresponding text prompt and preceding image-caption pairs; (ii) to address the data shortage of visual storytelling, we collect paired image-text sequences by sourcing from online videos and open-source E-books, establishing processing pipeline for constructing a large-scale dataset with diverse characters, storylines, and artistic styles, named StorySalon; (iii) Quantitative experiments and human evaluations have validated the superiority of our StoryGen, where we show StoryGen can generalize to unseen characters without any optimization, and generate image sequences with coherent content and consistent character. Code, dataset, and models are available at https://haoningwu3639.github.io/StoryGen_Webpage/

Figures (17)

• Figure 1: An illustration of open-ended visual storytelling. In practice, users can feed a unique and engaging story synthesized by a large language model into our proposed StoryGen model to generate a sequence of images coherently, denoted as open-ended visual story generation. And they can also provide a pre-defined character with its corresponding storyline, to perform open-ended visual story continuation. We recommend the reader to zoom in and read the story.
• Figure 2: Architecture Overview. (a) Our StoryGen model utilizes current text prompt and previous visual-language contexts as conditions to generate an image, iteratively synthesizing a coherent image sequence. Note the parameters of the corresponding attention layers are shared between Diffusion UNet and StoryGen. To avoid potential ambiguity, the parameters are not shared across UNet blocks in a single model. (b) The proposed Visual-Language Context Module can effectively combine the information from current text prompt and contexts from preceding image-caption pairs. (c) We add more noise to reference frames with longer temporal distances to the current frame as positional encoding to distinguish the temporal order. The multiple features can then be directly concatenated to serve as context conditions.
• Figure 3: Dataset Pipeline and Visualization. Left: Metadata sourced from the Internet undergoes a three-step pipeline including frame extraction, visual-language alignment and post-processing, resulting in properly aligned image-text pairs. Right: Our StorySalon dataset contains diverse styles and characters.
• Figure 4: Qualitative Comparison with other methods. The image sequences in orange, green, and blue boxes are generated by Prompt-SDM, AR-LDM and StoryGen respectively. Our synthesis results exhibit impressive performance superiority in terms of style, content and character consistency, text-image alignment, and image quality. Please refer to the Appendix for more qualitative results.
• Figure 5: Ablation studies on consistency. We incorporate our proposed Visual-Language Context Module into a pre-trained SDM, and train it on MS-COCO lin2014microsoft with other parameters frozen. The content consistency of single-object and multi-object generation on COCO and real data has demonstrated the effectiveness of our module. Please refer to the Appendix for experiment details and quantitative results.