TDVE-Assessor: Benchmarking and Evaluating the Quality of Text-Driven Video Editing with LMMs

TL;DR

This work tackles the challenge of evaluating text-driven video editing by introducing TDVE-DB, a large, multi-dimensional benchmark with 3,857 edited videos and 173,565 subjective ratings across edited video quality, editing alignment, and structural consistency. It also presents TDVE-Assessor, an LMM-based VQA system that fuses spatial/temporal video features with text prompts and uses a two-stage training regime with a LoRA-based regression head to deliver both qualitative feedback and quantitative scores. The approach achieves state-of-the-art correlations with human judgments on TDVE-DB and generalizes across additional benchmarks, highlighting its robustness for guiding future TDVE research. These contributions provide a rigorous, scalable framework for assessing and improving text-driven video edits, with potential impact on safer, more reliable AI-generated video workflows.

Abstract

Text-driven video editing is rapidly advancing, yet its rigorous evaluation remains challenging due to the absence of dedicated video quality assessment (VQA) models capable of discerning the nuances of editing quality. To address this critical gap, we introduce TDVE-DB, a large-scale benchmark dataset for text-driven video editing. TDVE-DB consists of 3,857 edited videos generated from 12 diverse models across 8 editing categories, and is annotated with 173,565 human subjective ratings along three crucial dimensions, i.e., edited video quality, editing alignment, and structural consistency. Based on TDVE-DB, we first conduct a comprehensive evaluation for the 12 state-of-the-art editing models revealing the strengths and weaknesses of current video techniques, and then benchmark existing VQA methods in the context of text-driven video editing evaluation. Building on these insights, we propose TDVE-Assessor, a novel VQA model specifically designed for text-driven video editing assessment. TDVE-Assessor integrates both spatial and temporal video features into a large language model (LLM) for rich contextual understanding to provide comprehensive quality assessment. Extensive experiments demonstrate that TDVE-Assessor substantially outperforms existing VQA models on TDVE-DB across all three evaluation dimensions, setting a new state-of-the-art. Both TDVE-DB and TDVE-Assessor will be released upon the publication.

Figures (14)

• Figure 1: (a) Acquisition of the source video (b) Generation of prompt words (c) Obtaining 170K subjective scores through subjective experiments (d) The number of videos for different models and different editing categories. (e) Three-dimensional scatter plot of subjective scores
• Figure 2: Bar charts of the score distribution of each model and editing category across three dimensions.
• Figure 3: Radar charts of the score distribution of each model and editing category across three dimensions.
• Figure 4: Scatter plots of score distribution between pairwise dimensions.
• Figure 5: (a) Video features are extracted by a frozen spatiotemporal encoder and aligned to a Large Language Model (Qwen2.5-VL-7B) using trainable projection modules. The LLM, fine-tuned with LoRA, then generates textual feedback for multiple evaluation dimensions via a trainable text decoder, while its last hidden states (quality representations) are input to a quality regression module. The model also supports pairwise comparison fine-tuning. (b) The quality regression module, a trainable MLP, converts these LLM-derived quality representations into numerical scores for dimensions like video quality, editing alignment, and structural consistency.