VideoMemory: Toward Consistent Video Generation via Memory Integration

TL;DR

VideoMemory tackles the problem of maintaining consistent entity identity across long-form narrative videos by introducing a Dynamic Memory Bank that stores explicit visual and semantic states for characters, props, and backgrounds. A three-agent pipeline—Storyboard, Memory, and Visualization—drives script-to-video generation with a retrieve-and-update memory mechanism that preserves identity while allowing story-driven changes. The approach is validated on a 54-case multi-shot consistency benchmark, where VideoMemory achieves superior cross-shot coherence and perceptual quality compared to baselines, with ablations confirming the essential role of memory. This memory-centric framework enables more reliable long-form video generation from textual prompts, with potential impacts on planning-driven video production and cinematic pre-visualization.

Abstract

Maintaining consistent characters, props, and environments across multiple shots is a central challenge in narrative video generation. Existing models can produce high-quality short clips but often fail to preserve entity identity and appearance when scenes change or when entities reappear after long temporal gaps. We present VideoMemory, an entity-centric framework that integrates narrative planning with visual generation through a Dynamic Memory Bank. Given a structured script, a multi-agent system decomposes the narrative into shots, retrieves entity representations from memory, and synthesizes keyframes and videos conditioned on these retrieved states. The Dynamic Memory Bank stores explicit visual and semantic descriptors for characters, props, and backgrounds, and is updated after each shot to reflect story-driven changes while preserving identity. This retrieval-update mechanism enables consistent portrayal of entities across distant shots and supports coherent long-form generation. To evaluate this setting, we construct a 54-case multi-shot consistency benchmark covering character-, prop-, and background-persistent scenarios. Extensive experiments show that VideoMemory achieves strong entity-level coherence and high perceptual quality across diverse narrative sequences.

Figures (8)

• Figure 1: From a single story prompt, VideoMemory generates coherent multi-shot videos using dynamic Character, Prop, and Background Memory Banks. This ensures exceptional entity consistency, as seen with the feather prop remaining perfectly stable across distant shots (e.g., 2, 10, and 12) despite significant scene and viewpoint variations.
• Figure 2: Comparison between existing methods and our method. (a) Existing methods lack entity-level memory, resulting in inconsistent appearances across shots. (b) VideoMemory retrieves and updates an entity-aware Dynamic Memory Bank during generation, enabling consistent multi-shot videos from text alone.
• Figure 3: The framework of the proposed VideoMemory. Starting from a script synopsis, our system plans shot-level descriptions, interacts with a Dynamic Memory Bank to retrieve or create entity references, generates keyframes, and finally synthesizes a coherent multi-shot video.
• Figure 4: An example "Harry Potter" workflow of VideoMemory. Starting from a textual synopsis, VideoMemory plans all scenes and shots without any visual input, incrementally builds the Dynamic Memory Bank while generating each shot, and finally composes a multi-shot sequence in which characters, props, and environments remain consistent over time (e.g., in Shot 7, Harry and Hermione at different ages are rendered coherently together with that shot's props and background).
• Figure 5: Qualitative comparison demonstrating superior entity consistency. Across all three subclasses (Character, Prop, Background), VideoMemory (bottom row) maintains remarkable stability where baselines fail. Note how baselines exhibit severe identity drift—changing a character's appearance (left), morphing a red kite into other objects (middle), and altering a garage's layout (right). In contrast, our method preserves the identity of all entities across distant shots, a direct result of our explicit memory management.