VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation

TL;DR

VEGGIE tackles instructional video editing by unifying grounding, reasoning, and pixel-level edits within a single end-to-end diffusion framework conditioned by a multimodal LLM. It uses frame-wise grounded task tokens and a lightweight alignment module to translate language-space instructions into pixel-space edits, trained with a two-stage curriculum from images to videos. A novel data synthesis pipeline expands high-quality instructional data from image edits into dynamic videos, complemented by VEG-Bench, a benchmark spanning eight editing skills. Empirical results show VEGGIE surpasses specialized baselines across diverse tasks, demonstrates zero-shot multimodal instructional following and few-shot in-context editing, and highlights beneficial cross-task learning among editing, grounding, and reasoning tasks.

Abstract

Recent video diffusion models have enhanced video editing, but it remains challenging to handle instructional editing and diverse tasks (e.g., adding, removing, changing) within a unified framework. In this paper, we introduce VEGGIE, a Video Editor with Grounded Generation from Instructions, a simple end-to-end framework that unifies video concept editing, grounding, and reasoning based on diverse user instructions. Specifically, given a video and text query, VEGGIE first utilizes an MLLM to interpret user intentions in instructions and ground them to the video contexts, generating frame-specific grounded task queries for pixel-space responses. A diffusion model then renders these plans and generates edited videos that align with user intent. To support diverse tasks and complex instructions, we employ a curriculum learning strategy: first aligning the MLLM and video diffusion model with large-scale instructional image editing data, followed by end-to-end fine-tuning on high-quality multitask video data. Additionally, we introduce a novel data synthesis pipeline to generate paired instructional video editing data for model training. It transforms static image data into diverse, high-quality video editing samples by leveraging Image-to-Video models to inject dynamics. VEGGIE shows strong performance in instructional video editing with different editing skills, outperforming the best instructional baseline as a versatile model, while other models struggle with multi-tasking. VEGGIE also excels in video object grounding and reasoning segmentation, where other baselines fail. We further reveal how the multiple tasks help each other and highlight promising applications like zero-shot multimodal instructional and in-context video editing.

Figures (17)

• Figure 1: We propose VEGGIE, a unified and versatile video generative model that handles various tasks for both video concept grounding and editing according to user instructions. With VEGGIE, users can locate, add, delete, and change concepts in a given video through diverse instruction formats (direct referring instruction or reasoning-demanding questions). Users can also edit videos with multimodal instruction empowered by MLLM, enabling applications like video editing from a reference image.
• Figure 2: Multimodal instruction following emerges in VEGGIE, allowing for style transfer or object addition from reference images.
• Figure 3: In-context editing emerges in VEGGIE, allowing for few-shot learning of editing tasks with paired image demonstrations.
• Figure 4: Overview of our proposed end-to-end VEGGIE framework. Our Multimodal Large Language Model first understands input video frames and diverse user instructions, then it generates frame-wise reasoning queries that maintain per-frame editing conditions for the video diffusion model. The video diffusion model will render the MLLM-generated conditions to the pixel space for diverse tasks, including video editing, video grounding, and video reasoning segmentation with questions. We only apply diffusion loss for the whole pipeline training.
• Figure 5: Our data generation pipeline for synthetic instructional video editing data. It injects dynamics into well-constructed instructional image editing datasets via the Image-to-Video (I2V) Model, and generates paired video data for instruction editing.