OmniV2V: Versatile Video Generation and Editing via Dynamic Content Manipulation

TL;DR

<3-5 sentence high-level summary> OmniV2V tackles the limitation of task-specific video generation/editing models by introducing a unified framework that combines a unified dynamic content manipulation injection module with a VLLM-driven instruction-based editing component. It fuses multi-modal conditioning (image, mask, pose) through a latent fusion tokenizer and 3D-VAE-based encoding, and aligns visual content with textual instructions via LLaVA, supported by a comprehensive multi-task dataset and OmniV2V-Test benchmark. Empirical results demonstrate competitive or superior performance across mask-guided editing, instruction-based edits, and controllable character video synthesis, often surpassing open-source baselines and approaching commercial methods. The work provides practical tooling and resources to advance cross-task video generation and editing research while highlighting considerations for responsible deployment.

Abstract

The emergence of Diffusion Transformers (DiT) has brought significant advancements to video generation, especially in text-to-video and image-to-video tasks. Although video generation is widely applied in various fields, most existing models are limited to single scenarios and cannot perform diverse video generation and editing through dynamic content manipulation. We propose OmniV2V, a video model capable of generating and editing videos across different scenarios based on various operations, including: object movement, object addition, mask-guided video edit, try-on, inpainting, outpainting, human animation, and controllable character video synthesis. We explore a unified dynamic content manipulation injection module, which effectively integrates the requirements of the above tasks. In addition, we design a visual-text instruction module based on LLaVA, enabling the model to effectively understand the correspondence between visual content and instructions. Furthermore, we build a comprehensive multi-task data processing system. Since there is data overlap among various tasks, this system can efficiently provide data augmentation. Using this system, we construct a multi-type, multi-scenario OmniV2V dataset and its corresponding OmniV2V-Test benchmark. Extensive experiments show that OmniV2V works as well as, and sometimes better than, the best existing open-source and commercial models for many video generation and editing tasks.

Figures (13)

• Figure 1: OmniV2V comprehensive capability demonstration. We showcase the excellent generation and editing results of OmniV2V, with the original input and the generated videos for each task displayed in the figure.
• Figure 2: The framework of OmniV2V. It consists of two main modules: a unified information injection module for integrating task requirements and a visual-text instruction module for understanding visual-instruction correspondence.
• Figure 3: Three types of strategies for injecting pose information.
• Figure 4: Qualitative of comparison on the wild dataset.
• Figure 5: Visualization of videos generated by OmniV2V on the wild dataset.