EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning

TL;DR

EditVerse introduces a unified transformer framework that unifies image and video editing and generation by representing text, image, and video as an interleaved token sequence. It leverages full self-attention and a novel four-dimensional Rotary Positional Embedding to support in-context learning and cross-modal knowledge transfer, enabling arbitrary resolution and duration outputs. A scalable data pipeline combines image and video data with a curated 232K video-editing dataset and a new EditVerseBench for evaluation, achieving state-of-the-art results against open-source and commercial models. The work highlights emergent abilities, showing cross-modal transfer reduces video data scarcity and suggests future directions in efficiency and specialization trade-offs.

Abstract

Recent advances in foundation models highlight a clear trend toward unification and scaling, showing emergent capabilities across diverse domains. While image generation and editing have rapidly transitioned from task-specific to unified frameworks, video generation and editing remain fragmented due to architectural limitations and data scarcity. In this work, we introduce EditVerse, a unified framework for image and video generation and editing within a single model. By representing all modalities, i.e., text, image, and video, as a unified token sequence, EditVerse leverages self-attention to achieve robust in-context learning, natural cross-modal knowledge transfer, and flexible handling of inputs and outputs with arbitrary resolutions and durations. To address the lack of video editing training data, we design a scalable data pipeline that curates 232K video editing samples and combines them with large-scale image and video datasets for joint training. Furthermore, we present EditVerseBench, the first benchmark for instruction-based video editing covering diverse tasks and resolutions. Extensive experiments and user studies demonstrate that EditVerse achieves state-of-the-art performance, surpassing existing open-source and commercial models, while exhibiting emergent editing and generation abilities across modalities.

Figures (9)

• Figure 1: The strong video editing performance of EditVerse emerges from a unified framework trained on a diverse set of mixed image and video data. This teaser visualizes a selection of supported image and video editing tasks (Full instructions are shown in the Appendix).
• Figure 2: Overview of EditVerse. We design a unified framework for image and video editing and generation, which processes text and vision inputs into a unified sequence. The right part of the figure shows our positional embedding design. This framework leverages full self-attention to facilitate robust in-context learning and effective knowledge transfer among modalities.
• Figure 3: Examples for the interleaved text and vision pattern. EditVerse is capable of processing image and video inputs and outputs of arbitrary resolution, duration, and sequential positions.
• Figure 4: Examples from the proposed EditVerseBench. EditVerseBench includes $200$ editing pairs, evenly distributed across 20 editing categories as well as horizontal and vertical orientations.
• Figure 5: User study on EditVerseBench.