FluencyVE: Marrying Temporal-Aware Mamba with Bypass Attention for Video Editing

TL;DR

FluencyVE tackles the challenge of one-shot video editing with diffusion models by integrating a temporal-aware Mamba module to enable dense global frame attention with linear complexity, and introducing Bypass Attention to reduce parameter count via low-rank projections. The approach preserves the generative power of pretrained Stable Diffusion while significantly cutting training and inference costs, addressing temporal inconsistency and computational overhead common in prior methods. Through extensive experiments and ablations on frame consistency and textual alignment, FluencyVE demonstrates improved editing quality across backgrounds, styles, and subjects with substantial efficiency gains. This yields a practical, scalable path for video editing via diffusion models, with potential extensions toward adapter-based fine-tuning to further reduce training requirements.

Abstract

Large-scale text-to-image diffusion models have achieved unprecedented success in image generation and editing. However, extending this success to video editing remains challenging. Recent video editing efforts have adapted pretrained text-to-image models by adding temporal attention mechanisms to handle video tasks. Unfortunately, these methods continue to suffer from temporal inconsistency issues and high computational overheads. In this study, we propose FluencyVE, which is a simple yet effective one-shot video editing approach. FluencyVE integrates the linear time-series module, Mamba, into a video editing model based on pretrained Stable Diffusion models, replacing the temporal attention layer. This enables global frame-level attention while reducing the computational costs. In addition, we employ low-rank approximation matrices to replace the query and key weight matrices in the causal attention, and use a weighted averaging technique during training to update the attention scores. This approach significantly preserves the generative power of the text-to-image model while effectively reducing the computational burden. Experiments and analyses demonstrate promising results in editing various attributes, subjects, and locations in real-world videos.

Figures (9)

• Figure 1: Illustration of the proposed FluencyVE for one-shot video editing. Given a text-video pair, we approximate the weight matrices $Q$ and $K$ in the original sparse-causal attention layer as $Q^{\prime}$ and $K^{\prime}$, and calculate new attention scores using a weighted average to reduce parameter count and training cost, subsequently fine-tuning only $Q^{\prime}$ and $K^{\prime}$.We also introduce a time-aware linear sequence module, TA-Mamba (Fig. \ref{['fig:padding']}(b)), to further enhance temporal awareness of video features, enabling smooth and continuous video editing effects. During fine-tuning, we follow the fine-tuning strategy from Tune-A-Video, updating only the weights of $Q$ in the cross attention layer. During inference, we sample a novel video from the latent noise, which is inverted from the input video and guided by an edited prompt.
• Figure 2: Different Scan Methods. Following the Spatial-First rule, we introduce four novel scanning methods by reversing temporal or spatial ordering.
• Figure 3: Illustration of the proposed Temporal-aware Mamba. (a) Unique trainable embedding vectors are assigned to each frame, enabling the model to better capture the temporal characteristics and intra-frame distributions. The sequence is then fed as the spatial-temporal forward input into (b) the temporal-aware Mamba , where flip operations generate inputs that follow the four-directional scanning strategy shown in Fig. \ref{['fig:scan']}, which are then processed by the SSM.
• Figure 4: Illustration of the Bypass Attention. Latent features of frame $v_{i}$, previous frames $v_{i-1}$ and $v_{1}$ are projected to the weight matrices, query $W_{Q}$, key $W_{K}$ and value $W_{V}$. We substitute the low-rank approximation matrices $W_q$ and $W_k$ for $W_{Q}$ and $W_{K}$, respectively, and compute the attention scores through weighted averaging.
• Figure 5: Video editing results from various input videos and prompts. Our model produces temporally consistent videos that accurately follow text prompts while preserving the original frame structure.