Beyond Rigid: Benchmarking Non-Rigid Video Editing

TL;DR

The paper tackles the lack of physics-aware evaluation for non-rigid video editing by introducing NRVBench, a dataset of 180 physics-grounded videos across six categories with 2,340 editing prompts and 360 QA items, plus NRVE-Acc, a Vision-Language Model–based metric that jointly measures instruction adherence, physical plausibility, and temporal coherence. To address evaluation gaps, it also presents VM-Edit, a training-free baseline that uses dual-region, clocked diffusion sampling to balance foreground plasticity with background stability. Through extensive experiments, existing methods struggle with non-rigid deformation realism, while VM-Edit demonstrates strong structure preservation, competitive text alignment, and solid temporal consistency, validating the benchmark's discriminatory power. The work offers a standardized framework for physics-aware non-rigid video editing and points to future directions in automatic parameter scheduling and more nuanced region-based editing strategies, aiming to accelerate progress in realistic motion-aware video editing.

Abstract

Despite the remarkable progress in text-driven video editing, generating coherent non-rigid deformations remains a critical challenge, often plagued by physical distortion and temporal flicker. To bridge this gap, we propose NRVBench, the first dedicated and comprehensive benchmark designed to evaluate non-rigid video editing. First, we curate a high-quality dataset consisting of 180 non-rigid motion videos from six physics-based categories, equipped with 2,340 fine-grained task instructions and 360 multiple-choice questions. Second, we propose NRVE-Acc, a novel evaluation metric based on Vision-Language Models that can rigorously assess physical compliance, temporal consistency, and instruction alignment, overcoming the limitations of general metrics in capturing complex dynamics. Third, we introduce a training-free baseline, VM-Edit, which utilizes a dual-region denoising mechanism to achieve structure-aware control, balancing structural preservation and dynamic deformation. Extensive experiments demonstrate that while current methods have shortcomings in maintaining physical plausibility, our method achieves excellent performance across both standard and proposed metrics. We believe the benchmark could serve as a standard testing platform for advancing physics-aware video editing.

Figures (21)

• Figure 1: Overview of NRVBench. The left side is the framework of NRVBench, which contains 180 physics-based non-rigid motion videos across six categories, paired with 2,340 fine-grained editing instructions and 360 multiple-choice questions. The right side is the traditional metrics results of six video editing models.
• Figure 2: Overview of the NRVE-Acc Evaluation Pipeline. Our framework utilizes standardized source/target prompt pairs to guide the editing models. We use GPT-4o to generate detailed annotations. The generated videos are then evaluated by Qwen2.5-VL using a hierarchical Question-Answer(QA) mechanism, focusing on (a) Instruction Following, (b) Temporal Consistency, and (c) Material-Specific Deformation.
• Figure 3: Overview of the VM-Edit model.
• Figure 4: Comparison of six models across six non-rigid categories using Motion Fidelity (scaled by $10^{2}$).
• Figure 5: NRVE-Acc results(Left) and human evaluation(Right) on CTS. Detailed results are provided in the appendices D.