IMAGEdit: Let Any Subject Transform

TL;DR

This work tackles the challenge of editing videos with arbitrary numbers of designated subjects without finetuning. It introduces IMAGEdit, a modular, training-free framework that combines prompt-guided multimodal alignment, prior-based mask retargeting, and a mask-driven video generator to produce faithful, temporally coherent edits while preserving non-target regions. The authors also construct MSVBench, a multi-subject benchmark, and demonstrate state-of-the-art performance across diverse, crowded scenes, while enabling plug-in compatibility with any mask-driven generator. They release code, models, and datasets to support replication and broader adoption in research and applications.

Abstract

In this paper, we present IMAGEdit, a training-free framework for any number of video subject editing that manipulates the appearances of multiple designated subjects while preserving non-target regions, without finetuning or retraining. We achieve this by providing robust multimodal conditioning and precise mask sequences through a prompt-guided multimodal alignment module and a prior-based mask retargeting module. We first leverage large models' understanding and generation capabilities to produce multimodal information and mask motion sequences for multiple subjects across various types. Then, the obtained prior mask sequences are fed into a pretrained mask-driven video generation model to synthesize the edited video. With strong generalization capability, IMAGEdit remedies insufficient prompt-side multimodal conditioning and overcomes mask boundary entanglement in videos with any number of subjects, thereby significantly expanding the applicability of video editing. More importantly, IMAGEdit is compatible with any mask-driven video generation model, significantly improving overall performance. Extensive experiments on our newly constructed multi-subject benchmark MSVBench verify that IMAGEdit consistently surpasses state-of-the-art methods. Code, models, and datasets are publicly available at https://github.com/XWH-A/IMAGEdit.

Figures (16)

• Figure 1: Visualization results of IMAGEdit. Given any video with any number of designated subjects, IMAGEdit performs precise category transformations while maintaining subject count and spatial layout. Especially in crowded scenes with overlapping subjects, IMAGEdit demonstrates stable consistent editing.
• Figure 2: Visual results generated from current video editing methods and our IMAGEdit (Dogs $\rightarrow$ Robot Wolves). Previous methods apparently retain the reference dog's appearance. In contrast, the result of IMAGEdit both aligns the robot wolf's features and captures the reference dog's layout.
• Figure 3: The IMAGEdit framework first derives robust multimodal cues via a prompt-guided multimodal alignment. Then, a prior-based mask retargeting module produces a time-consistent mask sequence aligned with the input video. Finally, the multimodal cues and mask sequence are fed into a video generation model to synthesize the edited video.
• Figure 4: Visualization of the without (w/o) and with (w/) multimodal condition. The first row: Hockey Players $\rightarrow$ Astronauts; the second row: Horse Riders $\rightarrow$ Gokus.
• Figure 5: Illustration of prompt-guided multimodal alignment. We generate aligned extended text conditions and extended image conditions for each original prompt.