PipeFlow: Pipelined Processing and Motion-Aware Frame Selection for Long-Form Video Editing

TL;DR

PipeFlow tackles the prohibitive computational cost of diffusion-based long-form video editing by introducing motion-aware frame skipping, a pipelined DDIM inversion and editing workflow, and neural frame interpolation to bridge segment boundaries. By partitioning videos into segments and overlapping preprocessing with editing across multiple GPUs, PipeFlow achieves near-linear scaling in runtime with video length and delivers substantial speedups (up to ~31.7× over DMT and ~9.6× over TokenFlow) while preserving temporal coherence and prompt fidelity. The method demonstrates strong empirical performance across short- and long-form videos, improved CLIP alignment (mean CLIP score near 97.5) and reduced warp error, and robust qualitative results against state-of-the-art baselines. Overall, PipeFlow offers a practical, scalable solution for diffusion-based long-form video editing with strong multi-GPU efficiency and minimal quality loss.

Abstract

Long-form video editing poses unique challenges due to the exponential increase in the computational cost from joint editing and Denoising Diffusion Implicit Models (DDIM) inversion across extended sequences. To address these limitations, we propose PipeFlow, a scalable, pipelined video editing method that introduces three key innovations: First, based on a motion analysis using Structural Similarity Index Measure (SSIM) and Optical Flow, we identify and propose to skip editing of frames with low motion. Second, we propose a pipelined task scheduling algorithm that splits a video into multiple segments and performs DDIM inversion and joint editing in parallel based on available GPU memory. Lastly, we leverage a neural network-based interpolation technique to smooth out the border frames between segments and interpolate the previously skipped frames. Our method uniquely scales to longer videos by dividing them into smaller segments, allowing PipeFlow's editing time to increase linearly with video length. In principle, this enables editing of infinitely long videos without the growing per-frame computational overhead encountered by other methods. PipeFlow achieves up to a 9.6X speedup compared to TokenFlow and a 31.7X speedup over Diffusion Motion Transfer (DMT).

Figures (5)

• Figure 1: Single-GPU Latency Comparison.A: Number of frames generated and the time needed to generate them for various methods. PipeFlow achieves the best performance compared to other state-of-the-art methods, achieving up to a 31.7$\times$ speedup compared to DMT. B: Comparison of the time needed to generate 240 frames with the average CLIP score of generated frames. PipeFlow achieves the best latency to CLIP Score tradeoff.
• Figure 2: PipeFlow Overview. The input video is analyzed for motion between frames (Section \ref{['subsec:motion_detection']}). Frames with low motion are skipped during editing and later interpolated. The video is then divided into N segments for editing. Each segment undergoes DDIM inversion as preprocessing. Once the noisy video segments are created, the keyframes are sampled for joint editing. After the first preprocessing pass, joint editing for segment i is performed in parallel with preprocessing for segment i+1, continuing until all N segments are processed.
• Figure 3: Qualitative comparisons against competing state-of-the-art video editing methods. As shown, ControlVideo zhang2023controlvideo suffers from inconsistent character generation. TokenFlow geyer2023tokenflow shows better temporal consistency, but suffers from weaker edits not corresponding to the prompt as well. StreamV2V streamv2v fails to follow the prompt properly and also suffers from larger amounts of flickering and motion blurring. Text2Video-Zero text2video-zero is unable to edit longer form video and also fails to properly maintain the motion of the original video. PipeFlow (ours) achieves the best visual quality while maintaining consistency throughout the videos.
• Figure 4: CLIP Score Vs. Warp Error Comparison. We present a comparison of the CLIP Score and Warp Error of PipeFlow vs. competing state-of-the-art methods. Our PipeFlow achieves the best CLIP Score and Warp Error against all competing methods.
• Figure 5: Ablation Study. Ablation study on object editing and style transfer prompts. PipeFlow achieves strong visual quality, while maintaining consistency throughout the videos.