In-Context Learning with Unpaired Clips for Instruction-based Video Editing

TL;DR

This work tackles data scarcity in instruction-based video editing by introducing a two-stage training strategy: pretrain on unpaired video clips to learn basic editing operations, then fine-tune on a compact, high-quality set of editing pairs. Built on HunyuanVideoT2V, the approach uses in-context inputs that incorporate the original video and a noised edit sequence, guided by editing instructions. The method achieves state-of-the-art performance in both instruction alignment and visual fidelity, requiring significantly fewer supervised examples (about 150K editing pairs) than traditional large-scale datasets. This data-efficient paradigm offers a practical path toward robust, instruction-driven video editing without relying on massive synthetic datasets.

Abstract

Despite the rapid progress of instruction-based image editing, its extension to video remains underexplored, primarily due to the prohibitive cost and complexity of constructing large-scale paired video editing datasets. To address this challenge, we introduce a low-cost pretraining strategy for instruction-based video editing that leverages in-context learning from unpaired video clips. We show that pretraining a foundation video generation model with this strategy endows it with general editing capabilities, such as adding, replacing, or deleting operations, according to input editing instructions. The pretrained model can then be efficiently refined with a small amount of high-quality paired editing data. Built upon HunyuanVideoT2V, our framework first pretrains on approximately 1M real video clips to learn basic editing concepts, and subsequently fine-tunes on fewer than 150k curated editing pairs to extend more editing tasks and improve the editing quality. Comparative experiments show that our method surpasses existing instruction-based video editing approaches in both instruction alignment and visual fidelity, achieving a 12\% improvement in editing instruction following and a 15\% improvement in editing quality.

Figures (8)

• Figure 1: Pipeline of training data curation. (a) shows the pipeline for curating clip data from raw videos; (b) illustrates the pipeline for synthesizing and filtering editing data.
• Figure 2: In-context Instruction-based Video Editing Model. The instruction, original video, and edited video are injected into the model in an in-context manner. The timesteps corresponding to the original video tokens are fixed to 0, while the timesteps of text and edited video tokens retain T.
• Figure 3: Comparison results with the instruction-based methods. For methods that require the first edited frame, the first frame is obtained using Qwen-Image-Edit wu2025qwen.
• Figure 4: Model performance with varying training steps and data. Dashed lines indicate evaluation results after continued training. The darker line in each sub-figure shows the pretrained model, which acquires basic editing capabilities from clip data, while the darker dashed line after 40K steps illustrates that prolonged pretraining causes model degradation. The lighter line demonstrates that the model improves rapidly with only a few SFT steps, and the lighter dashed line indicates that about 12K SFT steps are sufficient to fine-tune a high-quality editing model.
• Figure 5: Visualization results of the pretrained model. (a) shows that excessive pretraining leads to overfitting and collapse on unseen editing types. However, with a small number of SFT steps on editing data, the pretrained model quickly learns new editing types. (b) demonstrates that the pretrained model acquires basic editing operations, such as addition, removal, and replacement, using only video clip data. The left are the original videos, and the right are the edited results.