Visual Prompting for One-shot Controllable Video Editing without Inversion

TL;DR

This work tackles OCVE without relying on DDIM inversion by reframing editing as a visual-prompting task. It leverages a 2×2 grid input to an image inpainting diffusion model and introduces Content Consistency Sampling (CCS) to preserve content fidelity, along with Temporal-content Consistency Sampling (TCS) based on Stein Variational Gradient Descent to enforce temporal coherence. The method achieves state-of-the-art results on a MagicBrush-based dataset, offering improved edit fidelity, source faithfulness, and temporal smoothness with higher efficiency than previous OCVE approaches. By removing inversion and exploiting consistency-model-inspired sampling, the approach provides a practical, fast, and reliable OCVE solution with broad applicability to controllable video editing tasks.

Abstract

One-shot controllable video editing (OCVE) is an important yet challenging task, aiming to propagate user edits that are made -- using any image editing tool -- on the first frame of a video to all subsequent frames, while ensuring content consistency between edited frames and source frames. To achieve this, prior methods employ DDIM inversion to transform source frames into latent noise, which is then fed into a pre-trained diffusion model, conditioned on the user-edited first frame, to generate the edited video. However, the DDIM inversion process accumulates errors, which hinder the latent noise from accurately reconstructing the source frames, ultimately compromising content consistency in the generated edited frames. To overcome it, our method eliminates the need for DDIM inversion by performing OCVE through a novel perspective based on visual prompting. Furthermore, inspired by consistency models that can perform multi-step consistency sampling to generate a sequence of content-consistent images, we propose a content consistency sampling (CCS) to ensure content consistency between the generated edited frames and the source frames. Moreover, we introduce a temporal-content consistency sampling (TCS) based on Stein Variational Gradient Descent to ensure temporal consistency across the edited frames. Extensive experiments validate the effectiveness of our approach.

Figures (4)

• Figure 1: Visual prompting and one-shot controllable video editing share the goal of propagating certain modifications across images. In visual prompting, modifications made in the example (e.g., changing the color of the mountain from golden to green) are transferred to the query, whereas in one-shot controllable video editing, modifications applied to the first edited frame are propagated to the subsequent source frames.
• Figure 2: The overall pipeline of our method. First, to enable the image inpainting diffusion model to perform OCVE through visual prompting, we organize the example ($A$ and $A'$), query ($B$), and output ($B'$) from the visual prompting setup into a 2$\times$2 square grid, which serves as the input information (\ref{['subsec:Performing our task via visual prompting']}) to the inpainting diffusion model. Next, we modify the sampling process of the inpainting diffusion model, and design a content consistency sampling (\ref{['subsec:consistency sampling']}), to generate $\boldsymbol{B'}$ using the multi-step consistency sampling of the consistency models song2023consistency. Finally, based on the generated $\boldsymbol{B'}$, we apply Temporal-content Consistency Sampling (\ref{['subsec:SVGD']}) with Stein Variational Gradient Descent liu2016stein to adjust the source frames, enhancing their temporal consistency and yielding the final edited frames in our framework.
• Figure 3: A visualization of the input information $G(i)$ ($i$ denotes the $i$-th frame) and its corresponding mask information $M$.
• Figure 4: The visual comparison includes our method alongside two SOTA OCVE methods (AnyV2V ku2024anyv2v and Videoshop fan2024videoshop), evaluated across two distinct types of editing. On the left, user modifications consist of replacing the fruit in the basin with vegetables. On the right, the user edits involve: 1) removing the individual's hair and 2) adding glasses.