RFDM: Residual Flow Diffusion Model for Efficient Causal Video Editing

TL;DR

RFDM tackles the challenge of text-guided video editing with fixed-length inputs by introducing a causal, autoregressive framework that repurposes a 2D image-to-image diffusion backbone for video-to-video editing. A key innovation is the Residual Flow forward diffusion, which shifts the sampling mean toward the previous frame prediction and models a residual between consecutive frames, enabling faithful, temporally coherent edits with compute comparable to image models. Trained on the Señorita dataset and evaluated against TGVE, TGVE+, and Señorita benchmarks, RFDM achieves strong faithfulness and temporal consistency, often outperforming 2D baselines and rivaling 3D backbones while delivering lower latency and memory usage. The work also proposes a new benchmark and evaluation protocol to better capture edit faithfulness and temporal coherence, underscoring the practical potential of causal I2I backbones for scalable, real-time video editing.

Abstract

Instructional video editing applies edits to an input video using only text prompts, enabling intuitive natural-language control. Despite rapid progress, most methods still require fixed-length inputs and substantial compute. Meanwhile, autoregressive video generation enables efficient variable-length synthesis, yet remains under-explored for video editing. We introduce a causal, efficient video editing model that edits variable-length videos frame by frame. For efficiency, we start from a 2D image-to-image (I2I) diffusion model and adapt it to video-to-video (V2V) editing by conditioning the edit at time step t on the model's prediction at t-1. To leverage videos' temporal redundancy, we propose a new I2I diffusion forward process formulation that encourages the model to predict the residual between the target output and the previous prediction. We call this Residual Flow Diffusion Model (RFDM), which focuses the denoising process on changes between consecutive frames. Moreover, we propose a new benchmark that better ranks state-of-the-art methods for editing tasks. Trained on paired video data for global/local style transfer and object removal, RFDM surpasses I2I-based methods and competes with fully spatiotemporal (3D) V2V models, while matching the compute of image models and scaling independently of input video length. More content can be found in: https://smsd75.github.io/RFDM_page/

Figures (5)

• Figure 1: Video style transfer. Editing video style requires understanding of motion and style across frames. Naive Image-to-Image(I2I) models generate inconsistent video frames. Fairy smoothens the result, trading computational cost to achieve lower jittering and inconsistency. RFDM results in the most consistent edit, while being computationally similar to the per-frame baseline (I2I). The video is taken from the DAVIS.
• Figure 2: Training. We obtain the noisy input($y_t^s$) at time frame $t$ by fusing current frame($x_t$), previous prediction($\hat{y}_{t-1}$), and noise through a noise scheduler via Eq. \ref{['eq:forward_001']}. For a given video clip, we edit the first frame, where the noisy input is obtained by adding noise to the target; where the first target is $\hat{y}_{-1}=0$. Thereafter, the output on the previous frame is used as input ($\hat{y}_1$). The amount of noise, i.e. $\alpha^s, \sigma^s$, is sampled independently for each frame. This figure shows a scenario where the amount of noise is low in the first frame, i.e. $\alpha^s \rightarrow 1$, and high in the second frame, i.e. $\alpha^s \rightarrow 0$. In the latter, where the amount of noise is high, $y_t^s$ is close to the noisy $\hat{y}_{t-1}$. In addition to $y_t^s$, the denoising model takes $x_t$ and $\hat{y}_{t-1}$ as extra inputs via concatenation and the instruction prompt via cross attention.
• Figure 3: Inference illustration. We edit the frame at time $t$ by starting from the noisy previous frame ($\hat{y}_{t-1}$), and applying the denoising steps. The first frame, where $y_{-1}=0$, is edited in an I2I manner, starting from pure noise. In each denoising step, the current input frame($x_t$) and the clean previous prediction($\hat{y}_{t-1}$) are given to the model as extra input via concatenation to the noisy input. The instruction prompt is provided via cross-attention.
• Figure 4: Qualitative results. We compare RFDM with Fairy and VidToMe on Señorita benchmark across object removal (green), local style transfer (red), and global style transfer (blue). VidToMe is shown only for the style-transfer tasks as it is not designed for object removal. Compared to Fairy, RFDM yields more consistent and higher-fidelity outputs, especially in object removal where Fairy often leaves noticeable artifacts or alters irrelevant regions. VidToMe produces temporally consistent videos but deviates more from the ground truth than both RFDM and Fairy.
• Figure 5: Different styles generated by RFDM on Señorita.RFDM produces consistent and faithful outputs across a wide range of styles and input videos.