GenVideo: One-shot Target-image and Shape Aware Video Editing using T2I Diffusion Models

TL;DR

GenVideo tackles the limitation of text-only diffusion-based video editing by introducing target-image conditioning and shape-aware localization. It finetunes an inflated SD-unCLIP model on the source video, generates target-aware InvEdit masks via DDIM inversion, and applies a latent-correction strategy to maintain temporal consistency across frames. The approach yields superior target-text and target-image alignment while handling edits where the target object differs in shape or size from the source, as demonstrated against multiple baselines and through ablations. This method enables precise, temporally coherent edits in videos with practical impact for product-specific or appearance-driven video customization, while highlighting avenues for further improvement in fine-grained motion changes.

Abstract

Video editing methods based on diffusion models that rely solely on a text prompt for the edit are hindered by the limited expressive power of text prompts. Thus, incorporating a reference target image as a visual guide becomes desirable for precise control over edit. Also, most existing methods struggle to accurately edit a video when the shape and size of the object in the target image differ from the source object. To address these challenges, we propose "GenVideo" for editing videos leveraging target-image aware T2I models. Our approach handles edits with target objects of varying shapes and sizes while maintaining the temporal consistency of the edit using our novel target and shape aware InvEdit masks. Further, we propose a novel target-image aware latent noise correction strategy during inference to improve the temporal consistency of the edits. Experimental analyses indicate that GenVideo can effectively handle edits with objects of varying shapes, where existing approaches fail.

Figures (13)

• Figure 1: Overview of GenVideo. Inflated attention layers are finetuned during source video finetuning. During inference, InvEdit predicts a region to edit and latent correction uses that mask to improve the inter-frame temporal consistency. $\mathcal{M}_{\phi}$ - "no mask".
• Figure 2: InvEdit approach - the mask is generated by first iteratively computing noise differences across multiple timesteps for the source denoising branch and target denoising branch. Then, these differences are averaged and binarized to obtain the InvEdit mask.
• Figure 3: Latent correction strategy corrects the latent noise using UNet features from the previous and successive video frame. Correspondence Error (CE) map computed using the ground truth (source video correspondences) shows that the DDIM o/p of the model before correction has high CE (E.) which our latent correction strategy fixes (F.) using Up-block-2 feature correspondences which have low CE (D.).
• Figure 4: Source videos, Target images, InvEdit mask and the GenVideo results. Top: "A person wearing a gray black shoe.", and "A man panda rides a kite surfboard in deep waters." Bottom: "man naruto skiing on snow", and " man Tom Cruise walking down the street".
• Figure 5: Results of InvEdit on zero-shot image editing showing its capability for target-objects of varying shape and size.