Bridging Your Imagination with Audio-Video Generation via a Unified Director

TL;DR

UniMAGE proposes a unified director model that jointly handles script drafting and keyframe generation for long-form, multi-shot video creation. Built on a Mixture-of-Transformers framework, it combines Interleaved Concept Learning and Disentangled Expert Learning to fuse narrative reasoning with stable visual synthesis, aided by In-Context ID Prompting and Pre-Context Script Splitting. The approach yields state-of-the-art open-source performance in long-form coherence, character identity preservation, and visual consistency, as demonstrated on qualitative analyses, ViStoryBench metrics, and user studies. This work enables non-experts to generate cinematic sequences with coherent storytelling and aligned visuals, advancing automated film creation and multimodal generation research.

Abstract

Existing AI-driven video creation systems typically treat script drafting and key-shot design as two disjoint tasks: the former relies on large language models, while the latter depends on image generation models. We argue that these two tasks should be unified within a single framework, as logical reasoning and imaginative thinking are both fundamental qualities of a film director. In this work, we propose UniMAGE, a unified director model that bridges user prompts with well-structured scripts, thereby empowering non-experts to produce long-context, multi-shot films by leveraging existing audio-video generation models. To achieve this, we employ the Mixture-of-Transformers architecture that unifies text and image generation. To further enhance narrative logic and keyframe consistency, we introduce a ``first interleaving, then disentangling'' training paradigm. Specifically, we first perform Interleaved Concept Learning, which utilizes interleaved text-image data to foster the model's deeper understanding and imaginative interpretation of scripts. We then conduct Disentangled Expert Learning, which decouples script writing from keyframe generation, enabling greater flexibility and creativity in storytelling. Extensive experiments demonstrate that UniMAGE achieves state-of-the-art performance among open-source models, generating logically coherent video scripts and visually consistent keyframe images.

Figures (38)

• Figure 1: Showcase of UniMAGE’s multimodal directing abilities. UniMAGE unifies script drafting, extension, continuation, and keyframe image generation, thereby enabling coherent long-form storytelling with consistent characters and cinematic visual compositions. The generated scripts and keyframes can further serve as structured, high-level guidance for existing audio-video joint generation models.
• Figure 2: Script structure of UniMAGE. The script structure includes three components: global descriptions ($\mathcal{G}$), content descriptions ($\mathcal{C}$), and keyframe images ($\mathcal{F}$), together with a user prompt ($\rho$). Special tokens and indicator symbols are used to denote key script elements.
• Figure 3: Illustrations of Interleaved Concept Learning and Disentangled Expert Learning. To enhance visual consistency and logical coherence across long-context scripts, as well as to fully leverage both textual and image data, we first optimize all MoT parameters using interleaved text–image data, and then disentangle the training of the understanding and generation experts—using pure text scripts for the former and text–image data for the latter.
• Figure 4: Illustrations of In-Context ID Prompting and Pre-Context Script Splitting. The former enhances visual consistency by aligning generated images with global character and scene descriptions, while the latter enables adaptive narrative extension and continuation.
• Figure 5: Comparison with the baselines for multi-character script generation. UniMAGE demonstrates a superior ability to maintain consistent character identities and visual coherence across multiple shots, whereas the baseline methods fail to preserve such consistency.