CoAgent: Collaborative Planning and Consistency Agent for Coherent Video Generation

TL;DR

CoAgent addresses the brittleness of open-domain video generation by reframing synthesis as a closed-loop, multi-agent process that combines planning, explicit cross-shot memory, and verification. The Storyboard Planner decomposes inputs into structured shot plans, while the Global Context Manager preserves entity identity across shots, and the Verifier Agent provides feedback to trigger selective regeneration. The integration yields substantial improvements in narrative coherence, visual consistency, and text–video alignment compared with both agent-based baselines and foundation-model backbones, validated on VBench and VideoScore. The framework’s modular design facilitates controllability and scalability, with potential extensions to audio and multi-modal coherence for immersive video storytelling.

Abstract

Maintaining narrative coherence and visual consistency remains a central challenge in open-domain video generation. Existing text-to-video models often treat each shot independently, resulting in identity drift, scene inconsistency, and unstable temporal structure. We propose CoAgent, a collaborative and closed-loop framework for coherent video generation that formulates the process as a plan-synthesize-verify pipeline. Given a user prompt, style reference, and pacing constraints, a Storyboard Planner decomposes the input into structured shot-level plans with explicit entities, spatial relations, and temporal cues. A Global Context Manager maintains entity-level memory to preserve appearance and identity consistency across shots. Each shot is then generated by a Synthesis Module under the guidance of a Visual Consistency Controller, while a Verifier Agent evaluates intermediate results using vision-language reasoning and triggers selective regeneration when inconsistencies are detected. Finally, a pacing-aware editor refines temporal rhythm and transitions to match the desired narrative flow. Extensive experiments demonstrate that CoAgent significantly improves coherence, visual consistency, and narrative quality in long-form video generation.

Figures (6)

• Figure 1: The overall architecture of our framework.
• Figure 2: The detailed architecture of our CoAgent workflow. The framework integrates four key modules: the Storyboard Planner ($\mathcal{A}_{plan}$), the Global Context Manager ($\mathcal{M}_{GCM}$), the Synthesis Module ($\mathcal{A}_{synth}$), and the Verifier Agent ($\mathcal{A}_{verify}$).
• Figure 3: Case Study 1: GCM for Identity Preservation. For the prompt "A day of a Computer Science student, in vlog style," the Baseline (top row) exhibits severe identity drift, generating four different people. Our CoAgent (bottom row), empowered by the GCM, maintains a perfectly consistent identity across all shots, visually demonstrating the quantitative gains in Subject Consistency from Table \ref{['tab:ablation_study']}.
• Figure 4: Case Study 2: Verifier-driven Multi-Modal Correction. Our Verifier robustly corrects diverse errors identified in open-loop generation (top row of each pair). (a) Semantic Correction: The baseline defaults to common sense, while our Verifier (bottom) enforces the anti-common sense prompt. (b) Attribute Correction: The baseline ignores "blonde" while Ours (bottom) corrects the factual error. (c) Artifact Correction: The baseline produces temporal artifacts, while Ours (bottom) regenerates a stable shot.
• Figure 5: Continuous Frame Comparison (Prompt: "A panda standing on a surfboard in the ocean in sunset"). We extract 5 frames from the generated videos. Top Row (Sora2): Shows high fidelity but requires proprietary access. Bottom Row (CoAgent): Our open-source framework maintains robust subject identity (the panda) and consistent lighting interaction with the environment, matching the temporal coherence of the commercial state-of-the-art.