MM-StoryAgent: Immersive Narrated Storybook Video Generation with a Multi-Agent Paradigm across Text, Image and Audio

TL;DR

MM-StoryAgent tackles automated narrated storybook video generation across text, image, and audio by deploying a multi-agent, multi-stage pipeline that first crafts an attractive story and then produces modality-specific assets (images, speech, sound effects, and music) before composing them into a cohesive video. The framework emphasizes modularity and openness, enabling plug-and-play replacement of components and providing a topic-based evaluation dataset. Objective and subjective evaluations show notable gains in textual quality and cross-modal alignment over a direct prompting baseline, while highlighting areas for improvement in audio timing and cross-modal coherence. Overall, the work delivers a practical, extensible platform with potential impact on AI-generated educational and entertainment content for children.

Abstract

The rapid advancement of large language models (LLMs) and artificial intelligence-generated content (AIGC) has accelerated AI-native applications, such as AI-based storybooks that automate engaging story production for children. However, challenges remain in improving story attractiveness, enriching storytelling expressiveness, and developing open-source evaluation benchmarks and frameworks. Therefore, we propose and opensource MM-StoryAgent, which creates immersive narrated video storybooks with refined plots, role-consistent images, and multi-channel audio. MM-StoryAgent designs a multi-agent framework that employs LLMs and diverse expert tools (generative models and APIs) across several modalities to produce expressive storytelling videos. The framework enhances story attractiveness through a multi-stage writing pipeline. In addition, it improves the immersive storytelling experience by integrating sound effects with visual, music and narrative assets. MM-StoryAgent offers a flexible, open-source platform for further development, where generative modules can be substituted. Both objective and subjective evaluation regarding textual story quality and alignment between modalities validate the effectiveness of our proposed MM-StoryAgent system. The demo and source code are available.

Figures (3)

• Figure 1: MM-StoryAgent incorporates LLMs, generative models and tools to transform the input story setting into omni-modality storytelling videos.
• Figure 2: The overview of MM-StoryAgent framework. Agents discuss the writing requirements given story settings before the outline-chapter two-stage writing process. Then modality-specific assets are produced by generation tools and potential prompt revisers. Finally, the video compose agent produces the storytelling video.
• Figure 3: The gradio demonstration page.