Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices

Nathaniel Cohen, Vladimir Kulikov, Matan Kleiner, Inbar Huberman-Spiegelglas, Tomer Michaeli

TL;DR

Slicedit presents a zero-shot approach to video editing that leverages a pretrained text-to-image diffusion model by inflating it into a video denoiser. The core idea is to denoise not only frames but also spatiotemporal slices of the video volume, treating slices as natural-image-like inputs to enforce temporal coherence. By combining an extended-attention stream with a spatiotemporal denoising stream and performing volume-level DDPM inversion, Slicedit edits target regions while preserving unedited content and motion across long, nonrigid videos. Experiments show that Slicedit outperforms prior zero-shot methods in fidelity to the text prompt, structure preservation, and temporal consistency, with ablations validating the contributions of spatiotemporal slices and attention-injection. This yields a practical, scalable tool for text-based video editing without per-video fine-tuning, while acknowledging limitations in global style edits and potential societal misuse.

Abstract

Text-to-image (T2I) diffusion models achieve state-of-the-art results in image synthesis and editing. However, leveraging such pretrained models for video editing is considered a major challenge. Many existing works attempt to enforce temporal consistency in the edited video through explicit correspondence mechanisms, either in pixel space or between deep features. These methods, however, struggle with strong nonrigid motion. In this paper, we introduce a fundamentally different approach, which is based on the observation that spatiotemporal slices of natural videos exhibit similar characteristics to natural images. Thus, the same T2I diffusion model that is normally used only as a prior on video frames, can also serve as a strong prior for enhancing temporal consistency by applying it on spatiotemporal slices. Based on this observation, we present Slicedit, a method for text-based video editing that utilizes a pretrained T2I diffusion model to process both spatial and spatiotemporal slices. Our method generates videos that retain the structure and motion of the original video while adhering to the target text. Through extensive experiments, we demonstrate Slicedit's ability to edit a wide range of real-world videos, confirming its clear advantages compared to existing competing methods. Webpage: https://matankleiner.github.io/slicedit/

Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices

TL;DR

Slicedit presents a zero-shot approach to video editing that leverages a pretrained text-to-image diffusion model by inflating it into a video denoiser. The core idea is to denoise not only frames but also spatiotemporal slices of the video volume, treating slices as natural-image-like inputs to enforce temporal coherence. By combining an extended-attention stream with a spatiotemporal denoising stream and performing volume-level DDPM inversion, Slicedit edits target regions while preserving unedited content and motion across long, nonrigid videos. Experiments show that Slicedit outperforms prior zero-shot methods in fidelity to the text prompt, structure preservation, and temporal consistency, with ablations validating the contributions of spatiotemporal slices and attention-injection. This yields a practical, scalable tool for text-based video editing without per-video fine-tuning, while acknowledging limitations in global style edits and potential societal misuse.

Abstract

Text-to-image (T2I) diffusion models achieve state-of-the-art results in image synthesis and editing. However, leveraging such pretrained models for video editing is considered a major challenge. Many existing works attempt to enforce temporal consistency in the edited video through explicit correspondence mechanisms, either in pixel space or between deep features. These methods, however, struggle with strong nonrigid motion. In this paper, we introduce a fundamentally different approach, which is based on the observation that spatiotemporal slices of natural videos exhibit similar characteristics to natural images. Thus, the same T2I diffusion model that is normally used only as a prior on video frames, can also serve as a strong prior for enhancing temporal consistency by applying it on spatiotemporal slices. Based on this observation, we present Slicedit, a method for text-based video editing that utilizes a pretrained T2I diffusion model to process both spatial and spatiotemporal slices. Our method generates videos that retain the structure and motion of the original video while adhering to the target text. Through extensive experiments, we demonstrate Slicedit's ability to edit a wide range of real-world videos, confirming its clear advantages compared to existing competing methods. Webpage: https://matankleiner.github.io/slicedit/
Paper Structure (41 sections, 6 equations, 18 figures, 4 tables, 2 algorithms)

This paper contains 41 sections, 6 equations, 18 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Tuning T2I models for video editing.
  4. T2I models for zero-shot video editing.
  5. Spatiotemporal slices of the space-time volume.
  6. Preliminaries
  7. Method
  8. Inflated Denoiser
  9. Extended attention.
  10. Spatiotemporal slices.
  11. Combined zero-shot video denoiser.
  12. Video Editing
  13. Experiments
  14. Implementation Details
  15. Comparisons
  16. ...and 26 more sections

Figures (18)

  • Figure 1: Diffusion model as a spatiotemporal slice prior. The left pane shows the $(\mathbf{x},\mathbf{y},t)$ space-time volume of a video. The middle pane shows $\mathbf{y}-t$ slices of natural videos. The right pane shows images generated by Stable Diffusion using several text-prompts. The generated images have similar characteristics to spatiotemporal slices of natural videos. This suggests that a pretrained text-to-image model can serve as a good prior for spatiotemporal video slices.
  • Figure 2: Applying a pretrained image denoiser to spatiotemporal slices. The plot shows the MSE obtained when applying a pretrained Stable Diffusion denoiser to noisy video frames, spatiotemporal slices and permuted frames (all in latent space). The MSE for spatiotemporal slices is comparable to (even lower than) the MSE for frames, and both are lower than the MSE for permuted frames, which are out-of-distribution for the denoiser.
  • Figure 3: Slicedit overview. Left: To edit a video $I_0$, we apply DDPM inversion using our video-denoising model, which is an inflated version of the T2I model. This process extracts noise volumes and attention maps for each diffusion timestep. Subsequently, we run DDPM sampling using the extracted noise space, while injecting the extended attention maps at specific timesteps. The inversion and sampling are performed while conditioning the inflated denoiser on the source and target text prompts, respectively. Right: Our inflated denoiser employs two versions of the pretrained image denoiser. A version with extended attention is applied to $\mathbf{x}-\mathbf{y}$ slices (blue), and the original denoiser is applied to $\mathbf{y}-t$ slices (red). The two predicted noise volumes are then combined into the final predicted noise volume (marked in green).
  • Figure 4: Slicedit Results. Our method edits only the specified regions of the input video according to the target prompt while keeping the unspecified regions the same. The output video maintains coherence between frames (i.e., the same robot, origami rabbit and cheetah across the frames). Video results are available on our https://matankleiner.github.io/slicedit/#examples
  • Figure 5: Qualitative comparison. We compare our method against other state-of-the-art zero-shot video editing methods. Our method edits only the specified region, according to the text prompt, and keeps the unspecified regions unchanged. The competing methods often change the entire frame, specified and unspecified regions alike. See videos on our https://matankleiner.github.io/slicedit/#comparisons
  • ...and 13 more figures