From Words to Worlds: Transforming One-line Prompt into Immersive Multi-modal Digital Stories with Communicative LLM Agent

TL;DR

StoryAgent addresses the challenge of producing long-duration, multi-modal digital stories by integrating communicative LLM agents with cutting-edge generative tools. It employs a top-down story drafting process and a bottom-up asset-generation pipeline organized as a Story Cluster of agent teams, underpinned by an expert-critic feedback loop to ensure cross-modal consistency. The framework enables scene interactivity in 2D/3D contexts, supports flexible human intervention, and does not rely on reference videos, enhancing scalability and adaptability. Together, these contributions offer a plug-and-play, extensible approach that can democratize content creation while keeping pace with rapidly evolving generative-model technologies.

Abstract

Digital storytelling, essential in entertainment, education, and marketing, faces challenges in production scalability and flexibility. The StoryAgent framework, introduced in this paper, utilizes Large Language Models and generative tools to automate and refine digital storytelling. Employing a top-down story drafting and bottom-up asset generation approach, StoryAgent tackles key issues such as manual intervention, interactive scene orchestration, and narrative consistency. This framework enables efficient production of interactive and consistent narratives across multiple modalities, democratizing content creation and enhancing engagement. Our results demonstrate the framework's capability to produce coherent digital stories without reference videos, marking a significant advancement in automated digital storytelling.

Figures (13)

• Figure 1: StoryAgent is a digital storytelling generation framework that integrates communicative Large Language Model agents with state-of-the-art generative models and tools. Taking one-line text instruction as input, it produces digital storytelling content with scene interactivity, long-duration consistency, and intervention flexibility.
• Figure 2: The framework of StoryAgent. Beginning with a text instruction, the framework builds the story with task decomposition, specifying all asset files for each modality in the textual description. Then generative models and tools are organized to create and compose tangible assets of the story.
• Figure 3: Structure of a two-stage LLM agent team. It takes upstream JSON as inputs and uses two expert-critic LLM agent pairs to process. Finally, another JSON string will be generated for downstream teams.
• Figure 4: Semantic scene understanding example
• Figure 5: Character pants asset examples