Q-Save: Towards Scoring and Attribution for Generated Video Evaluation

TL;DR

Q-Save addresses the need for unified, explainable evaluation of AI-generated videos by introducing a large-scale dataset (~10k videos) with MOS and fine-grained attributions across visual quality, dynamic quality, and text-video alignment, and by proposing a single model that jointly scores and explains across these dimensions. The framework combines SlowFast-based video processing, Chain-of-Thought prompts, and a three-stage training pipeline (SFT → GRPO → SFT) with a dual-supervision loss to achieve state-of-the-art instance- and model-level correlations ($SRCC$, $PLCC$) while delivering human-aligned justifications. Cross-dataset validation demonstrates strong generalization to out-of-domain benchmarks, underscoring the practical utility of unified, interpretable video evaluation. The work lays a foundation for explainable multimodal video evaluation and provides a pathway for community adoption through dataset and code release. Overall, Q-Save advances trustworthy AI in generative video by enabling precise quality assessment and transparent, rationale-based explanations.

Abstract

We present Q-Save, a new benchmark dataset and model for holistic and explainable evaluation of AI-generated video (AIGV) quality. The dataset contains near 10000 videos, each annotated with a scalar mean opinion score (MOS) and fine-grained attribution labels along three core dimensions: visual quality, dynamic quality, and text-video alignment. These multi-aspect annotations enable both accurate quality assessment and interpretable reasoning behind the scores. To leverage this data, we propose a unified evaluation model that jointly performs quality scoring and attribution-based explanation. The model adopts the SlowFast framework to distinguish between fast frames and slow frames - slow frames are processed with high resolution while fast frames use low resolution, balancing evaluation accuracy and computational efficiency. For training, we use data formatted in Chain-of-Thought (COT) style and employ a multi-stage strategy: we first conduct Supervised Fine-Tuning (SFT), then further enhance the model with Grouped Relative Policy Optimization (GRPO), and finally perform SFT again to improve model stability. Experimental results demonstrate that our model achieves state-of-the-art performance in video quality prediction while also providing human-aligned, interpretable justifications. Our dataset and model establish a strong foundation for explainable evaluation in generative video research, contributing to the development of multimodal generation and trustworthy AI. Code and dataset will be released upon publication.

Figures (11)

• Figure 1: Abilities of Q-Save-tuned Qwen3-VL-8B-Instruct on Score and Interpreter, in comparison with the baseline version.
• Figure 2: Human annotation process overview: 1) Raters are trained using text-defined rating levels; we simulate this with a rating-based syllabus for LMMs. 2) Raters assign scores and select reasons for low ratings (Bad–Fair). 3) Ratings are aggregated into MOS, and reasons are converted into descriptions; we propose a probability-based inference method for LMMs to generate final outputs.
• Figure 3: The model structure and video preprocessing. The model is based on the fine-tuned Qwen3-VL-8B-Instruct and adopts the SlowFast for video preprocessing to optimize the extraction of spatiotemporal features from videos (FPS = 8).
• Figure 4: Overview of the Q-Save construction process and training strategy. We design prompts and employ models to generate videos. Subjective evaluations are then conducted to rate the quality of these generated instances. The resulting scores and descriptions constitute a dataset, and the corresponding knowledge is then injected into the linear mixed model according to the specified training strategy.
• Figure 5: Distribution of Mean Opinion Scores (MOS) across six generative models (Veo2, Kling, Kling2.0, Hunyuan, Dreamina, Wan), evaluated along four quality dimensions: Text Alignment (TA), Motion Quality (MQ), Visual Quality (VQ), and Overall Quality (All). Each subplot represents a model-dimension pair, with MOS scores (1–5) on the x-axis and frequency counts (0–500) on the y-axis. Blue bars denote histogram counts, while overlaid curves represent smoothed kernel density estimates.