Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Temporally Consistent Object Editing in Videos using Extended Attention

AmirHossein Zamani, Amir G. Aghdam, Tiberiu Popa, Eugene Belilovsky

TL;DR

The paper tackles the challenge of localized, temporally coherent video editing. It introduces a framework that substitutes self-attention with Extended Attention in a pre-trained inpainting Stable Diffusion model, enabling cross-frame dependencies without fine-tuning and handling arbitrary mask shapes. By processing frames with a mask and text prompts over $T=50$ diffusion steps and using a frame-aware pre-processing stage, the method achieves consistent edits across frames for tasks such as object retargeting, replacement, and removal. Quantitative and qualitative results on BANMo, DAVIS, and YouTube-VOS demonstrate competitive fidelity and robust temporal coherence, with ablations validating key design choices and demonstrating memory-friendly modifications to the diffusion model.

Abstract

Image generation and editing have seen a great deal of advancements with the rise of large-scale diffusion models that allow user control of different modalities such as text, mask, depth maps, etc. However, controlled editing of videos still lags behind. Prior work in this area has focused on using 2D diffusion models to globally change the style of an existing video. On the other hand, in many practical applications, editing localized parts of the video is critical. In this work, we propose a method to edit videos using a pre-trained inpainting image diffusion model. We systematically redesign the forward path of the model by replacing the self-attention modules with an extended version of attention modules that creates frame-level dependencies. In this way, we ensure that the edited information will be consistent across all the video frames no matter what the shape and position of the masked area is. We qualitatively compare our results with state-of-the-art in terms of accuracy on several video editing tasks like object retargeting, object replacement, and object removal tasks. Simulations demonstrate the superior performance of the proposed strategy.

Temporally Consistent Object Editing in Videos using Extended Attention

TL;DR

The paper tackles the challenge of localized, temporally coherent video editing. It introduces a framework that substitutes self-attention with Extended Attention in a pre-trained inpainting Stable Diffusion model, enabling cross-frame dependencies without fine-tuning and handling arbitrary mask shapes. By processing frames with a mask and text prompts over diffusion steps and using a frame-aware pre-processing stage, the method achieves consistent edits across frames for tasks such as object retargeting, replacement, and removal. Quantitative and qualitative results on BANMo, DAVIS, and YouTube-VOS demonstrate competitive fidelity and robust temporal coherence, with ablations validating key design choices and demonstrating memory-friendly modifications to the diffusion model.

Abstract

Image generation and editing have seen a great deal of advancements with the rise of large-scale diffusion models that allow user control of different modalities such as text, mask, depth maps, etc. However, controlled editing of videos still lags behind. Prior work in this area has focused on using 2D diffusion models to globally change the style of an existing video. On the other hand, in many practical applications, editing localized parts of the video is critical. In this work, we propose a method to edit videos using a pre-trained inpainting image diffusion model. We systematically redesign the forward path of the model by replacing the self-attention modules with an extended version of attention modules that creates frame-level dependencies. In this way, we ensure that the edited information will be consistent across all the video frames no matter what the shape and position of the masked area is. We qualitatively compare our results with state-of-the-art in terms of accuracy on several video editing tasks like object retargeting, object replacement, and object removal tasks. Simulations demonstrate the superior performance of the proposed strategy.
Paper Structure (8 sections, 10 figures, 1 table)

This paper contains 8 sections, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experiments and Results
  4. Conclusions
  5. Quanitative Evaluation
  6. Ablation Studies
  7. Modifications to SD Model
  8. Additional Results

Figures (10)

  • Figure 1: An overview of the diffusion process for temporal consistent video editing. To make the video editing process temporally consistent, we extend the U-Net architecture by replacing original attention modules used in LDMs with Extended Attention modules
  • Figure 2: A qualitative results of the video object replacement task for different examples
  • Figure 3: A qualitative comparison between ProPainter ProPainter, E2FGVI E2FGVI, and our proposed method for the object removal task
  • Figure 4: A qualitative comparison between ProPainter ProPainter, E2FGVI E2FGVI, and our proposed method for the object retargeting task
  • Figure 5: Ablation studies on different components and mechanisms of the proposed method.
  • ...and 5 more figures