A Video is Worth 256 Bases: Spatial-Temporal Expectation-Maximization Inversion for Zero-Shot Video Editing

TL;DR

This paper tackles zero-shot diffusion-based video editing by addressing inaccurate frame-wise inversion and temporal inconsistency. It introduces Spatial-Temporal Expectation-Maximization (STEM) inversion, which learns a fixed, low-rank basis set ${\bm\mu}$ from the entire video and uses STEM-SA for all self-attention, reducing complexity from $\mathcal{O}(N(HW)^2)$ to $\mathcal{O}(NHWK)$ while improving reconstruction. The EM-based E-step and M-step—${\mathcal Z}$ responsibilities and basis updates—create a compact, global representation that enhances temporal coherence and editing performance when integrated with existing pipelines like TokenFlow and FateZero. Extensive experiments on DAVIS and web videos show improved PSNR/SSIM for reconstruction and CLIP-warp metrics for editing, with user studies confirming preferable results over state-of-the-art zero-shot methods. The approach offers a practical, non-finetuning inversion that can bolster diffusion-based video editing in real-world scenarios.

Abstract

This paper presents a video inversion approach for zero-shot video editing, which models the input video with low-rank representation during the inversion process. The existing video editing methods usually apply the typical 2D DDIM inversion or naive spatial-temporal DDIM inversion before editing, which leverages time-varying representation for each frame to derive noisy latent. Unlike most existing approaches, we propose a Spatial-Temporal Expectation-Maximization (STEM) inversion, which formulates the dense video feature under an expectation-maximization manner and iteratively estimates a more compact basis set to represent the whole video. Each frame applies the fixed and global representation for inversion, which is more friendly for temporal consistency during reconstruction and editing. Extensive qualitative and quantitative experiments demonstrate that our STEM inversion can achieve consistent improvement on two state-of-the-art video editing methods. Project page: https://stem-inv.github.io/page/.

Figures (10)

• Figure 1: The illustration of the proposed STEM inversion method. We estimate a more compact representation (bases $\bm{\mu}$) for the input video via the EM algorithm. The ST-E step and ST-M step are executed alternately for $R$ times until convergence. The Self-attention (SA) in our STEM inversion are denoted as STEM-SA, where the $\rm{Key}$ and $\rm{Value}$ embeddings are derived by projections of the converged $\bm{\mu}$.
• Figure 2: Ablation with different basis number $K$. Left: The reconstruction results of DDIM and our STEM inversion. Right: The corresponding editing results of various inversion settings, where TokenFlow editing process is used. Best viewed with zoom-in.
• Figure 3: Qualitative and quantitative comparison of the reconstruction with DDIM and STEM inversion, where two reconstruction fashions are applied: (i) DDIM reconstruction, (ii) DDIM reconstruction with additional attention fusion (i.e., Fatezero fatezero reconstruction).
• Figure 4: Qualitative comparison between different video editing methods. The editing scenarios here include style transfer, attribute editing, and shape editing. Best viewed with zoom-in.
• Figure 5: Visualization of different inversions. We first estimate optical flow raft of the input video. Then, we apply PCA to the features of last SA layer from the decoder with different inversion. Next, we use the optical flow to warp the features of former-frame to obtain the warped features in the 4-th column. Last, we show the cosine similarity of features from 3-rd and 4-th column.