DAPE: Dual-Stage Parameter-Efficient Fine-Tuning for Consistent Video Editing with Diffusion Models

TL;DR

This work tackles the challenge of editing videos with diffusion models while balancing quality and computational cost. It introduces DAPE, a dual-stage parameter-efficient fine-tuning framework comprising adjustable norm-tuning for temporal coherence and a visual adapter for enhanced appearance, designed to avoid negative interactions between the two stages. The authors also present the DAPE Dataset, a large-scale, high-quality benchmark with $232$ videos and $6$ editing prompts to enable objective, comprehensive evaluation. Extensive experiments on the DAPE Dataset and established benchmarks (BalanceCC, LOVEU-TGVE, RAVE) demonstrate state-of-the-art improvements in temporal coherence and text-video alignment, highlighting the practical potential of dual-stage PEFT for video editing.

Abstract

Video generation based on diffusion models presents a challenging multimodal task, with video editing emerging as a pivotal direction in this field. Recent video editing approaches primarily fall into two categories: training-required and training-free methods. While training-based methods incur high computational costs, training-free alternatives often yield suboptimal performance. To address these limitations, we propose DAPE, a high-quality yet cost-effective two-stage parameter-efficient fine-tuning (PEFT) framework for video editing. In the first stage, we design an efficient norm-tuning method to enhance temporal consistency in generated videos. The second stage introduces a vision-friendly adapter to improve visual quality. Additionally, we identify critical shortcomings in existing benchmarks, including limited category diversity, imbalanced object distribution, and inconsistent frame counts. To mitigate these issues, we curate a large dataset benchmark comprising 232 videos with rich annotations and 6 editing prompts, enabling objective and comprehensive evaluation of advanced methods. Extensive experiments on existing datasets (BalanceCC, LOVEU-TGVE, RAVE) and our proposed benchmark demonstrate that DAPE significantly improves temporal coherence and text-video alignment while outperforming previous state-of-the-art approaches.

Figures (10)

• Figure 1: Overall of DAPE. DAPE is based on the diffusion model. In the first stage, only the norm layers are fine-tuned. In the second stage, the visual adapter is inserted at specific positions for fine-tuning.
• Figure 2: Dataset statistics. Distributions of the DAPE Dataset across six semantic dimensions: category and complexity for subject, background, and event.
• Figure 3: Qualitative comparison. Different model performance on given video editing tasks. Our method achieves the best performance in terms of temporal consistency, text alignment and visual quality.
• Figure 4: User Study Results. Comparison of subjective scores for each model on Temporal Consistency, Text Alignment, and Overall Quality. Models performing the best, second best and third best scores 6, 5 and 4, and the scores for each model are weighted by vote frequency. Our model achieved the highest rating in all three aspects.
• Figure 5: Ablation of adapters. For better clarity, we index UNet blocks from ① to ⑦. The results are shown in Table \ref{['tab:adapter_ablation']}.