Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MUSE: A Multi-agent Framework for Unconstrained Story Envisioning via Closed-Loop Cognitive Orchestration

Wenzhang Sun, Zhenyu Wang, Zhangchi Hu, Chunfeng Wang, Hao Li, Wei Chen

TL;DR

MUSE reframes long-form audio-visual storytelling as a closed-loop constraint enforcement problem, bridging the intent–execution gap by explicitly encoding narrative intent as machine-executable constraints and iteratively refining multimodal outputs via plan–execute–verify–revise cycles. It introduces a modular, multi-agent architecture with identity anchoring, layout-aware synthesis, and temporal coherence mechanisms, coupled with a Vocal Trait Synthesis module for reference-free, consistent voice. The paper also presents MUSEBench, a reference-free, multi-dimensional evaluation protocol validated against human judgments, enabling holistic assessment of open-ended storytelling. Experimental results show substantial gains in long-horizon coherence, cross-modal identity consistency, and cinematic quality over representative baselines. This framework and its benchmark offer a scalable path toward controllable, auditable, and reproducible open-ended multimedia storytelling systems.

Abstract

Generating long-form audio-visual stories from a short user prompt remains challenging due to an intent-execution gap, where high-level narrative intent must be preserved across coherent, shot-level multimodal generation over long horizons. Existing approaches typically rely on feed-forward pipelines or prompt-only refinement, which often leads to semantic drift and identity inconsistency as sequences grow longer. We address this challenge by formulating storytelling as a closed-loop constraint enforcement problem and propose MUSE, a multi-agent framework that coordinates generation through an iterative plan-execute-verify-revise loop. MUSE translates narrative intent into explicit, machine-executable controls over identity, spatial composition, and temporal continuity, and applies targeted multimodal feedback to correct violations during generation. To evaluate open-ended storytelling without ground-truth references, we introduce MUSEBench, a reference-free evaluation protocol validated by human judgments. Experiments demonstrate that MUSE substantially improves long-horizon narrative coherence, cross-modal identity consistency, and cinematic quality compared with representative baselines.

MUSE: A Multi-agent Framework for Unconstrained Story Envisioning via Closed-Loop Cognitive Orchestration

TL;DR

MUSE reframes long-form audio-visual storytelling as a closed-loop constraint enforcement problem, bridging the intent–execution gap by explicitly encoding narrative intent as machine-executable constraints and iteratively refining multimodal outputs via plan–execute–verify–revise cycles. It introduces a modular, multi-agent architecture with identity anchoring, layout-aware synthesis, and temporal coherence mechanisms, coupled with a Vocal Trait Synthesis module for reference-free, consistent voice. The paper also presents MUSEBench, a reference-free, multi-dimensional evaluation protocol validated against human judgments, enabling holistic assessment of open-ended storytelling. Experimental results show substantial gains in long-horizon coherence, cross-modal identity consistency, and cinematic quality over representative baselines. This framework and its benchmark offer a scalable path toward controllable, auditable, and reproducible open-ended multimedia storytelling systems.

Abstract

Generating long-form audio-visual stories from a short user prompt remains challenging due to an intent-execution gap, where high-level narrative intent must be preserved across coherent, shot-level multimodal generation over long horizons. Existing approaches typically rely on feed-forward pipelines or prompt-only refinement, which often leads to semantic drift and identity inconsistency as sequences grow longer. We address this challenge by formulating storytelling as a closed-loop constraint enforcement problem and propose MUSE, a multi-agent framework that coordinates generation through an iterative plan-execute-verify-revise loop. MUSE translates narrative intent into explicit, machine-executable controls over identity, spatial composition, and temporal continuity, and applies targeted multimodal feedback to correct violations during generation. To evaluate open-ended storytelling without ground-truth references, we introduce MUSEBench, a reference-free evaluation protocol validated by human judgments. Experiments demonstrate that MUSE substantially improves long-horizon narrative coherence, cross-modal identity consistency, and cinematic quality compared with representative baselines.
Paper Structure (33 sections, 7 equations, 15 figures, 7 tables)

This paper contains 33 sections, 7 equations, 15 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Problem Definition
  5. Closed-Loop Omni-Modal Orchestration
  6. Script Decomposition and Identity Anchoring
  7. Planning: script structuring and identity state construction ($\Phi_{\text{pre}}$).
  8. Layout-Aware Multimodal Asset Synthesis
  9. Planning: routing and spatial control ($\Phi_{\text{prod}}$).
  10. Temporal Synthesis
  11. MUSEBench
  12. Motivation.
  13. Experiments
  14. Experimental Setup
  15. Quantitative Evaluation on ViStoryBench
  16. ...and 18 more sections

Figures (15)

  • Figure 1: With only simple text inputs, MUSE can generate storytelling videos of diverse styles with high continuity.
  • Figure 2: Overview of MUSE. Long-form audio-visual storytelling is realized through a closed-loop orchestration that coordinates specialist agents across identity (pre-production), space (production), and time (post-production).
  • Figure 3: MUSE can generate diverse character assets from text descriptions while ensuring cross audio-visual consistency and inter-character identity distinction.
  • Figure 4: Dynamic routing enables the generation of diverse camera-movement shots while preserving identity stability.
  • Figure 5: The feedback module evaluates generated scene assets from multiple perspectives and provides revision suggestions.
  • ...and 10 more figures