Memory-V2V: Augmenting Video-to-Video Diffusion Models with Memory

TL;DR

Memory-V2V tackles the problem of cross-turn consistency in multi-turn video editing by augmenting pretrained video-to-video diffusion models with an explicit visual memory. It combines an external cache of past edits with a retrieval mechanism (VideoFOV) and dynamic tokenization to condition each edit on the most relevant memories while aggressively controlling compute via adaptive token merging. The framework is extended to text-guided long video editing by segmenting long inputs and using memory-conditioned refinement, demonstrated on novel-view synthesis and long-form editing, with both qualitative and quantitative improvements over baselines. Results show significant efficiency gains (e.g., over 30% speedup from token compression and up to 90% FLOPs reduction from dynamic tokenization) and improved cross-iteration consistency without sacrificing single-turn performance, making iterative diffusion-based video editing more practical and reliable.

Abstract

Recent foundational video-to-video diffusion models have achieved impressive results in editing user provided videos by modifying appearance, motion, or camera movement. However, real-world video editing is often an iterative process, where users refine results across multiple rounds of interaction. In this multi-turn setting, current video editors struggle to maintain cross-consistency across sequential edits. In this work, we tackle, for the first time, the problem of cross-consistency in multi-turn video editing and introduce Memory-V2V, a simple, yet effective framework that augments existing video-to-video models with explicit memory. Given an external cache of previously edited videos, Memory-V2V employs accurate retrieval and dynamic tokenization strategies to condition the current editing step on prior results. To further mitigate redundancy and computational overhead, we propose a learnable token compressor within the DiT backbone that compresses redundant conditioning tokens while preserving essential visual cues, achieving an overall speedup of 30%. We validate Memory-V2V on challenging tasks including video novel view synthesis and text-conditioned long video editing. Extensive experiments show that Memory-V2V produces videos that are significantly more cross-consistent with minimal computational overhead, while maintaining or even improving task-specific performance over state-of-the-art baselines. Project page: https://dohunlee1.github.io/MemoryV2V

Figures (21)

• Figure 1: Memory-V2V enables iterative video-to-video editing with long-term visual memory, producing videos that remain consistent with all previously edited videos.(a) and (b) illustrate results for video novel view synthesis and text-guided long video editing, respectively. Colored boxes in (a) highlight novel-view regions that must remain consistent across generations. Note that each editing iteration performs independent denoising. Please refer to our project page for video results: https://dohunlee1.github.io/MemoryV2V/
• Figure 2: Overview of Memory-V2V. (a) From an external cache of previously edited videos, only the top-$k$ most relevant videos are retrieved and used as memory inputs to ensure cross-iteration consistency. (b) Dynamic tokenizers allocate more tokens to highly relevant videos—preserving fine details while maintaining an efficient overall token budget. (c) Adaptive token merging reduces latency and FLOPs by compressing less informative frames based on their attention-based responsiveness to the target query.
• Figure 3: Comparison of different memory encoders on two-turn novel view synthesis. The red-colored box depicts the novel region which are expected to be consistent between ${\boldsymbol x}_{1}$ and ${\boldsymbol x}_{2}$.
• Figure 4: Long-video editing as multi-turn video editing with Memory-V2V. (a) We extend target videos from existing video editing dataset for Memory-V2V training. (b) During inference, the external cache $\Omega$ stores the editing history as source–target video pairs. At the $i$-th editing turn, relevant memory videos are retrieved based on the similarity between source video segments.
• Figure 5: Qualitative results for multi-turn video novel view synthesis. Compared to baselines, Memory-V2V (Ours) generates videos from new camera trajectories while maintaining consistency across all previously generated novel regions (e.g., red and blue areas).