StoryMem: Multi-shot Long Video Storytelling with Memory

TL;DR

StoryMem tackles long-form, multi-shot video storytelling by introducing Memory-to-Video (M2V), a memory-augmented conditioning mechanism that repurposes pre-trained single-shot diffusion models for coherent minute-long narratives. It maintains a compact, dynamically updated keyframe memory and injects it into generation via latent concatenation and negative RoPE shifts, with LoRA-only fine-tuning. Memory extraction uses CLIP-based semantic keyframe selection and HPSv3 aesthetic filtering to ensure informative memories while a memory sink balances long- and short-term context. Evaluations on ST-Bench show superior cross-shot consistency and high visual fidelity compared to baselines, demonstrating the practical potential for scalable, coherent long-form storytelling.

Abstract

Visual storytelling requires generating multi-shot videos with cinematic quality and long-range consistency. Inspired by human memory, we propose StoryMem, a paradigm that reformulates long-form video storytelling as iterative shot synthesis conditioned on explicit visual memory, transforming pre-trained single-shot video diffusion models into multi-shot storytellers. This is achieved by a novel Memory-to-Video (M2V) design, which maintains a compact and dynamically updated memory bank of keyframes from historical generated shots. The stored memory is then injected into single-shot video diffusion models via latent concatenation and negative RoPE shifts with only LoRA fine-tuning. A semantic keyframe selection strategy, together with aesthetic preference filtering, further ensures informative and stable memory throughout generation. Moreover, the proposed framework naturally accommodates smooth shot transitions and customized story generation applications. To facilitate evaluation, we introduce ST-Bench, a diverse benchmark for multi-shot video storytelling. Extensive experiments demonstrate that StoryMem achieves superior cross-shot consistency over previous methods while preserving high aesthetic quality and prompt adherence, marking a significant step toward coherent minute-long video storytelling.

Figures (11)

• Figure 1: Given a story script with per-shot text descriptions, StoryMem generates appealing minute-long, multi-shot narrative videos with highly coherent characters and cinematic visual quality. This is achieved through shot-by-shot generation using a memory-conditioned single-shot video diffusion model.
• Figure 2: Overview of StoryMem. StoryMem generates each shot conditioned on a memory bank that stores keyframes from previously generated shots. During generation, the selected memory frames are encoded by a 3D VAE, fused with noisy video latents and binary masks, and fed into a LoRA-finetuned memory-conditioned Video DiT to synthesize the current shot. After generating each shot, semantic keyframe selection and aesthetic preference filtering are applied to obtain informative and reliable memory frames, enabling long-range cross-shot consistency and natural narrative progression. By iteratively generating shots with memory updates, StoryMem produces coherent minute-long, multi-shot story videos.
• Figure 3: Qualitative comparison. Our StoryMem generates coherent multi-scene, multi-shot story videos aligned with per-shot descriptions. In contrast, the pretrained model and keyframe-based baselines fail to preserve long-term character and scene consistency, while HoloCine meng2025holocine exhibits noticeable degradation in visual quality.
• Figure 4: User study. Our method is consistently preferred over all baselines in most dimensions, highlighting its superior multi-shot consistency and narrative coherence. Win indicates that users prefer our method over the baseline, Tie indicates no significant preference, and Lose indicates that users prefer the baseline.
• Figure 5: MR2V results. StoryMem enables customized story video generation by using reference images as the initial memory. The real-person reference images were used with proper consent from the individuals involved.