VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editing

TL;DR

The paper presents VIVID-10M, the first open large-scale dataset for video local editing, and a diffusion-based model, VIVID, capable of adding, modifying, and deleting entities within video regions. A key contribution is the KIVE mechanism, which guides interactive keyframe edits and propagates improvements to other frames, reducing latency and enabling efficient long-video editing. By training with a mix of image and video data and employing multi-task objectives, VIVID achieves state-of-the-art results on automated metrics and strong user-study preferences across addition, modification, and deletion tasks, while also reducing training overhead. The work offers a practical, interactive solution for real-world video editing and provides open access to the VIVID-10M dataset for further research.

Abstract

Diffusion-based image editing models have made remarkable progress in recent years. However, achieving high-quality video editing remains a significant challenge. One major hurdle is the absence of open-source, large-scale video editing datasets based on real-world data, as constructing such datasets is both time-consuming and costly. Moreover, video data requires a significantly larger number of tokens for representation, which substantially increases the training costs for video editing models. Lastly, current video editing models offer limited interactivity, often making it difficult for users to express their editing requirements effectively in a single attempt. To address these challenges, this paper introduces a dataset VIVID-10M and a baseline model VIVID. VIVID-10M is the first large-scale hybrid image-video local editing dataset aimed at reducing data construction and model training costs, which comprises 9.7M samples that encompass a wide range of video editing tasks. VIVID is a Versatile and Interactive VIdeo local eDiting model trained on VIVID-10M, which supports entity addition, modification, and deletion. At its core, a keyframe-guided interactive video editing mechanism is proposed, enabling users to iteratively edit keyframes and propagate it to other frames, thereby reducing latency in achieving desired outcomes. Extensive experimental evaluations show that our approach achieves state-of-the-art performance in video local editing, surpassing baseline methods in both automated metrics and user studies. The VIVID-10M dataset are open-sourced at https://kwaivgi.github.io/VIVID/.

Figures (12)

• Figure 1: Our method enables seamless addition, modification, and deletion of entities in videos. Edits are guided by masks and text, which specify both the target position and desired content.
• Figure 2: Compare VIVID-10M with existing image and video editing datasets.
• Figure 3: Each sample in VIVID-10M contains ground truth, masked data, masks and local caption.
• Figure 4: Data construction pipelines, where solid lines are required for both image and video data, and the dashed lines are only for video.
• Figure 5: Caption distribution. Each sample has three captions of different lengths.