Streaming Drag-Oriented Interactive Video Manipulation: Drag Anything, Anytime!

TL;DR

The work tackles streaming, fine-grained drag control for autoregressive video diffusion models by introducing REVEL and a training-free solution, DragStream. DragStream integrates Adaptive Distribution Self-Rectification (ADSR) to curb latent-space drift and Spatial-Frequency Selective Optimization (SFSO) to leverage context frames without overwhelming the generation with their signals, enabling editing and animation via drag operations in real time. The approach achieves superior streaming manipulation quality across metrics like FVD, FID, ObjMC, and DAI, and demonstrates compatibility with multiple VDM backbones, including CausVid, while avoiding costly model finetuning. This has practical implications for real-time, user-guided video editing and animation without heavy computational costs or retraining.

Abstract

Achieving streaming, fine-grained control over the outputs of autoregressive video diffusion models remains challenging, making it difficult to ensure that they consistently align with user expectations. To bridge this gap, we propose \textbf{stReaming drag-oriEnted interactiVe vidEo manipuLation (REVEL)}, a new task that enables users to modify generated videos \emph{anytime} on \emph{anything} via fine-grained, interactive drag. Beyond DragVideo and SG-I2V, REVEL unifies drag-style video manipulation as editing and animating video frames with both supporting user-specified translation, deformation, and rotation effects, making drag operations versatile. In resolving REVEL, we observe: \emph{i}) drag-induced perturbations accumulate in latent space, causing severe latent distribution drift that halts the drag process; \emph{ii}) streaming drag is easily disturbed by context frames, thereby yielding visually unnatural outcomes. We thus propose a training-free approach, \textbf{DragStream}, comprising: \emph{i}) an adaptive distribution self-rectification strategy that leverages neighboring frames' statistics to effectively constrain the drift of latent embeddings; \emph{ii}) a spatial-frequency selective optimization mechanism, allowing the model to fully exploit contextual information while mitigating its interference via selectively propagating visual cues along generation. Our method can be seamlessly integrated into existing autoregressive video diffusion models, and extensive experiments firmly demonstrate the effectiveness of our DragStream.

Figures (13)

• Figure 1: Examples of our REVEL task. The streaming video manipulation results shown above---including both Editing and Animation with drag effects such as object translation ("Trans"), deformation ("Defor"), and rotation ("Rot")---are produced by our DragStream method.
• Figure 2: Examples of Challenge \ref{['chal:1']} and Challenge \ref{['chal:2']}.
• Figure 3: Schematic illustration of our DragStream, where an Adaptive Distribution Self-Rectification (ADSR) strategy and a Spatial-Frequency Selective Optimization (SFSO) mechanism are designed to suppress latent distribution drift and context interference, respectively.
• Figure 4: Visualization results achieved by our DragStream on REVEL. Note that Editing produces only one video frame, but we insert an extra subsequent frame to maintain layout consistency with Animation.
• Figure 5: Quantitative performance achieved by our method in terms of ObjMC, FVD, FID, and DAI. "$\downarrow$’’ indicates that lower values correspond to better performance.

Theorems & Definitions (5)

• Definition 1: REVEL
• Proposition 1: Unifying Drag-Style Video Manipulation Operations
• Proposition 2: Adaptive Distribution Self-Rectification
• Proposition 3: Switchable Frequency-domain Selection
• Proposition 4: Criticality-driven Spatial-domain Selection