StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration

TL;DR

This work introduces StoryAgent, a multi-agent framework for Customized Storytelling Video Generation (CSVG) that decomposes the task into story design, storyboard creation, and animation. It introduces a three-stage workflow supported by a Storyboard Generator with an AnyDoor-inspired removal-redraw pipeline and a LoRA-BE customized Image-to-Video method to enforce inter- and intra-shot subject consistency. Extensive experiments on public datasets (PororoSV, FlintstonesSV) and open-domain subjects demonstrate superior subject fidelity and video quality compared with state-of-the-art baselines, validated by objective metrics and user studies. The framework's modularity and targeted consistency enhancements suggest strong practical impact for controllable storytelling video production, with noted limitations and clear directions for future improvements.

Abstract

The advent of AI-Generated Content (AIGC) has spurred research into automated video generation to streamline conventional processes. However, automating storytelling video production, particularly for customized narratives, remains challenging due to the complexity of maintaining subject consistency across shots. While existing approaches like Mora and AesopAgent integrate multiple agents for Story-to-Video (S2V) generation, they fall short in preserving protagonist consistency and supporting Customized Storytelling Video Generation (CSVG). To address these limitations, we propose StoryAgent, a multi-agent framework designed for CSVG. StoryAgent decomposes CSVG into distinct subtasks assigned to specialized agents, mirroring the professional production process. Notably, our framework includes agents for story design, storyboard generation, video creation, agent coordination, and result evaluation. Leveraging the strengths of different models, StoryAgent enhances control over the generation process, significantly improving character consistency. Specifically, we introduce a customized Image-to-Video (I2V) method, LoRA-BE, to enhance intra-shot temporal consistency, while a novel storyboard generation pipeline is proposed to maintain subject consistency across shots. Extensive experiments demonstrate the effectiveness of our approach in synthesizing highly consistent storytelling videos, outperforming state-of-the-art methods. Our contributions include the introduction of StoryAgent, a versatile framework for video generation tasks, and novel techniques for preserving protagonist consistency.

Figures (14)

• Figure 1: Comparison results of customized storytelling videos. Existing methods fail to preserve the subject consistency across shots, while our method successfully maintains inter-shot and intra-shot consistency of the customized subject.
• Figure 2: Our multi-agent framework's video creation process. Yellow blocks represent the next agent's input, while blue blocks indicate the current agent's output. For example, the Storyboard Generator (SG)'s input includes story results and reference videos, and its output consists of storyboard results and the subject mask of the reference videos. The Agent Manager (AM) automatically selects the next agent to execute upon receiving signals from different agents and may request the Observer to evaluate the results when other agents complete their tasks.
• Figure 3: The workflow diagrams of Storyboard Generator, along with the corresponding inputs (yellow blocks) and the outputs of their submodules (blue blocks).
• Figure 4: The illustration of our customized I2V generation method. Only the LoRA parameters inside each attention block and the block-wise token embeddings are trained to remember the subject. A localization loss is applied to enforce the tokens' cross-attention maps to focus on the subject.
• Figure 5: The Result visualization of three methods and the ground truth. The texts at the bottom are the story descriptions. The other two methods (the first 2 rows) fail to capture inter- and intra-shot consistency, our results (the $3_{rd}$ row) are more approaching the ground truth (the $4_{th}$ row).