VALA: Learning Latent Anchors for Training-Free and Temporally Consistent

TL;DR

VALA addresses the challenge of temporal inconsistency in training-free video editing by introducing a variational latent alignment module that compresses per-frame latents into a small set of semantic anchors. It learns adaptive, probabilistic frame-to-anchor assignments through a contrastive objective, enabling the preserved content and motion coherence across frames. The method can be integrated with existing T2I-based VE pipelines, achieving state-of-the-art inversion fidelity, editing quality, and temporal consistency while reducing memory and compute. Extensive experiments on DAVIS and diverse Internet videos demonstrate its superiority over prior training-free approaches, with ablations confirming the beneficial role of variational priors and anchor economy for robustness and efficiency.

Abstract

Recent advances in training-free video editing have enabled lightweight and precise cross-frame generation by leveraging pre-trained text-to-image diffusion models. However, existing methods often rely on heuristic frame selection to maintain temporal consistency during DDIM inversion, which introduces manual bias and reduces the scalability of end-to-end inference. In this paper, we propose~\textbf{VALA} (\textbf{V}ariational \textbf{A}lignment for \textbf{L}atent \textbf{A}nchors), a variational alignment module that adaptively selects key frames and compresses their latent features into semantic anchors for consistent video editing. To learn meaningful assignments, VALA propose a variational framework with a contrastive learning objective. Therefore, it can transform cross-frame latent representations into compressed latent anchors that preserve both content and temporal coherence. Our method can be fully integrated into training-free text-to-image based video editing models. Extensive experiments on real-world video editing benchmarks show that VALA achieves state-of-the-art performance in inversion fidelity, editing quality, and temporal consistency, while offering improved efficiency over prior methods.

Figures (4)

• Figure 1: Comparison between existing heuristic latent frame selection methods and our variational alignment approach. Upper: heuristic methods fix reference frames without learning. Bottom: our method VALA adaptively extracts semantic frame latents via variational alignment to have compact latent anchors.
• Figure 2: The training and inference process of VALA. Instead of relying on fixed alignment functions $\varphi(\cdot)$ in \ref{['eq:allign']}, VALA selects a compact and meaningful latent anchors to represent the spatiotemporal structure of a video learned by our posterior $f_{\phi}$. In the training stage, a lightweight network parameterized by $\phi$ is optimized by automatically learn the soft assignment matrix ${\boldsymbol{\mathbf{R}}}$. In the inference stage, the latent anchors ${\boldsymbol{\mathbf{C}}}$ is assigned by learned from soft assignment matrix and input latent representation ${\boldsymbol{\mathbf{Z}}}$ to support the following edition process.
• Figure 3: Qualitative comparison of editing results. Our variational inversion preserves temporal coherence and improves object fidelity under challenging prompts.
• Figure 4: Effect of the anchor number $A$ on inversion and editing performance. Performance improves with $A$, but corrupt beyond $A=512$. Left part is the reconstruction and right part is editing.