VINCIE: Unlocking In-context Image Editing from Video

TL;DR

This work introduces a scalable approach to annotate videos as interleaved multimodal sequences and designs a block-causal diffusion transformer trained on three proxy tasks: next-image prediction, current segmentation prediction, and next-segmentation prediction.

Abstract

In-context image editing aims to modify images based on a contextual sequence comprising text and previously generated images. Existing methods typically depend on task-specific pipelines and expert models (e.g., segmentation and inpainting) to curate training data. In this work, we explore whether an in-context image editing model can be learned directly from videos. We introduce a scalable approach to annotate videos as interleaved multimodal sequences. To effectively learn from this data, we design a block-causal diffusion transformer trained on three proxy tasks: next-image prediction, current segmentation prediction, and next-segmentation prediction. Additionally, we propose a novel multi-turn image editing benchmark to advance research in this area. Extensive experiments demonstrate that our model exhibits strong in-context image editing capabilities and achieves state-of-the-art results on two multi-turn image editing benchmarks. Despite being trained exclusively on videos, our model also shows promising abilities in multi-concept composition, story generation, and chain-of-editing applications.

Figures (29)

• Figure 1: By learning from videos, our method could attain universal in-context editing and generation abilities to handle various practical creation scenarios.
• Figure 2: Our session data construction pipeline. We use a VLM to annotate the visual transitions. We then use the generated textual descriptions to prompt GroundingDINO+SAM2, extracting segmentation masks for the edited regions.
• Figure 3: Model architecture. We apply a diffusion transformer framework (initialized from a video generative foundation model) with full attention to learn from the multimodal interleaved context, through three tasks (CSP, NSP, and NIP). Losses are only computed on noised tokens.
• Figure 4: Category distribution of MSE-Bench. "others" includes expression, orientation, position, global, and action change.
• Figure 5: Editing success rates in 5 turns at various data scales.