VIRES: Video Instance Repainting via Sketch and Text Guided Generation

TL;DR

VIRES addresses the challenge of temporally consistent video editing guided by sketches and text by integrating a Sequential ControlNet with standardized self-scaling, a sketch-attention augmented DiT backbone, and a sketch-aware encoder within a diffusion-based video generation framework. It introduces VireSet to train and evaluate video instance editing methods and demonstrates superior visual quality, temporal coherence, and alignment to user-provided sketches and descriptions compared with state-of-the-art approaches. The method supports repainting, replacement, generation, and removal of video instances, with additional capabilities such as sketch-to-video generation and sparse-sketch guidance. Despite higher computational cost, VIRES offers a flexible, controllable approach for high-fidelity video editing and provides extensive ablations and supplementary material to validate its components and applications.

Abstract

We introduce VIRES, a video instance repainting method with sketch and text guidance, enabling video instance repainting, replacement, generation, and removal. Existing approaches struggle with temporal consistency and accurate alignment with the provided sketch sequence. VIRES leverages the generative priors of text-to-video models to maintain temporal consistency and produce visually pleasing results. We propose the Sequential ControlNet with the standardized self-scaling, which effectively extracts structure layouts and adaptively captures high-contrast sketch details. We further augment the diffusion transformer backbone with the sketch attention to interpret and inject fine-grained sketch semantics. A sketch-aware encoder ensures that repainted results are aligned with the provided sketch sequence. Additionally, we contribute the VireSet, a dataset with detailed annotations tailored for training and evaluating video instance editing methods. Experimental results demonstrate the effectiveness of VIRES, which outperforms state-of-the-art methods in visual quality, temporal consistency, condition alignment, and human ratings. Project page: https://hjzheng.net/projects/VIRES/

Figures (11)

• Figure 1: Our VIRES model demonstrates powerful video editing capabilities with sketch and text guidance, as shown in four typical scenarios: (a) Repainting the color and style of the man's shirt. (b) Replacing the pickup truck with the dark-colored SUV. (c) Generating a running corgi within a video clip. (d) Removing a specified football from a video clip.
• Figure 2: The pipeline of VIRES model. (a) The input video clip and corresponding text descriptions are encoded using independent encoders. Noise is selectively added to the latent code according to the instance mask. (b) This noised latent code is fed into the denoising network, composed of staked transformer blocks with spatial and temporal self-attention modules, trained to reverse the diffusion process. (c) The Sequential ControlNet extracts structure layouts and injects them into the denoising network with the standardized self-scaling. (d) The sketch attention is implemented as a parallel branch within the spatial self-attention module, injecting structure semantics into subsequent transformer blocks. (e) A sketch-aware encoder additionally provides multi-level texture features during decoding, generating the repainted video clip from the denoised latent code.
• Figure 3: Visual quality comparisons of video editing methods with text and sketch guidance.
• Figure 4: Ablation study results of different VIRES variants.
• Figure 5: Examples of extended applications: (a) Generating an entire video clip with the sketch sequence. (b) Repainting the video instance with a single sketch frame.