AutoStory: Generating Diverse Storytelling Images with Minimal Human Effort

TL;DR

AutoStory tackles the challenge of generating diverse, high-quality storytelling images that align with narrative text while preserving consistent character identities. It couples large language model–driven layout planning with diffusion-based image synthesis, bridging sparse bounding-box layouts to dense guidance through a dense-condition generation module. The approach introduces training-free identity-consistency mechanisms for multi-view character generation and eliminates character-data collection by leveraging one-shot customization and 3D-aware priors. Experiments show strong text alignment, identity preservation, and visual quality, outperforming baselines in both text-only and character-fed settings. The work enables flexible, user-friendly story visualization across characters, scenes, and styles with minimal human effort.

Abstract

Story visualization aims to generate a series of images that match the story described in texts, and it requires the generated images to satisfy high quality, alignment with the text description, and consistency in character identities. Given the complexity of story visualization, existing methods drastically simplify the problem by considering only a few specific characters and scenarios, or requiring the users to provide per-image control conditions such as sketches. However, these simplifications render these methods incompetent for real applications. To this end, we propose an automated story visualization system that can effectively generate diverse, high-quality, and consistent sets of story images, with minimal human interactions. Specifically, we utilize the comprehension and planning capabilities of large language models for layout planning, and then leverage large-scale text-to-image models to generate sophisticated story images based on the layout. We empirically find that sparse control conditions, such as bounding boxes, are suitable for layout planning, while dense control conditions, e.g., sketches and keypoints, are suitable for generating high-quality image content. To obtain the best of both worlds, we devise a dense condition generation module to transform simple bounding box layouts into sketch or keypoint control conditions for final image generation, which not only improves the image quality but also allows easy and intuitive user interactions. In addition, we propose a simple yet effective method to generate multi-view consistent character images, eliminating the reliance on human labor to collect or draw character images.

Figures (14)

• Figure 1: The overall pipeline of our proposed method. The user only needs to provide a short command describing the story and optionally a few images for each character. The pipeline can be roughly divided into (a) the condition preparation stage, where we generate the bounding box layout with corresponding text prompts and the sketch or keypoint dense conditions, and (b) the conditional image generation stage, where we leverage a multi-subject customization model for story images generation, under the guidance of the prepared conditions. The story-to-layout and dense condition generation modules are detailed in (c) and (d), respectively. Specifically, we utilize the LLM for prompt and layout generation in (c) and leverage off-the-shelf perception models to extract dense control signals from object images generated by the single-subject customization model in (d). Both layouts and sketches are easy to understand and manipulate for user interactions.
• Figure 2: Identity-consistent character image generation. To generate multiple identity-consistent images of a single character in (c), we first generate a single character image, then apply a view-point conditioned image translation model to obtain the multi-view images in (a). Afterward, we extract the sketch conditions of those images in (b) and use them as conditions to improve the diversity of the final character image generation. A training-free consistency modeling method is introduced to improve identity consistency in (d).
• Figure 3: A few storytelling results. Texts below images are the plots of each panel. (a) and (b) are obtained with both user-provided story and character images, while (c) and (d) are obtained with only story text input. The user-provided or generated characters are presented in Appendix \ref{['subsec:app_details']}.
• Figure 4: Comparison with existing story visualization methods. The input characters are shown on the left. Note the results of TaleCrafter gong2023talecrafter are directly taken from their paper.
• Figure 5: Comparison with existing story visualization methods on the FlintstonesSV dataset. The input characters are shown on the left. Note the results of Make-A-Story rahman2022make and TaleCrafter gong2023talecrafter are directly taken from their paper.