Score2Instruct: Scaling Up Video Quality-Centric Instructions via Automated Dimension Scoring

Abstract

Classical video quality assessment methods generate a numerical score to judge a video's perceived visual fidelity and clarity. Yet, a score fails to describe the video's complex quality dimensions, restricting its applicability. Benefiting from the human-friendly linguistic output, adapting video large multimodal models to VQA via instruction tuning has the potential to address this issue. The core of the approach lies in the video quality-centric instruction data. Previous explorations mainly focus on the image domain, and their data generation processes heavily rely on human quality annotations and proprietary systems, limiting data scalability and effectiveness. To address these challenges, we propose the Score-based Instruction Generation pipeline. Specifically, SIG first scores multiple quality dimensions of an unlabeled video and maps scores to text-defined levels. It then explicitly incorporates a hierarchical Chain-of-Thought to model the correlation between specific dimensions and overall quality, mimicking the human visual system's reasoning process. The automated pipeline eliminates the reliance on expert-written quality descriptions and proprietary systems, ensuring data scalability and generation efficiency. To this end, the resulting Score2Instruct dataset contains over 320K diverse instruction-response pairs, laying the basis for instruction tuning. Moreover, to advance video LMMs' quality scoring and justification abilities simultaneously, we devise a progressive tuning strategy to fully unleash the power of S2I. Built upon SIG, we further curate a benchmark termed S2I-Bench with 400 open-ended questions to better evaluate the quality justification capacity of video LMMs. Experimental results on the S2I-Bench and existing benchmarks indicate that our method consistently improves quality scoring and justification capabilities across multiple video LMMs.

Figures (4)

• Figure 1: More visualized cases of S2I-Bench
• Figure 2: Overview of the Score-based Instruction Generation. It first samples more than 100K videos from VQA and general databases based on specific criteria. It then evaluates 14 quality dimensions to produce detailed dimension-wise ratings. Finally, a hierarchical chain-of-thought is applied to these ratings to derive the full justifications, while an LLM further expands the dataset with additional QA pairs.
• Figure 3: S2I comprises 320K instruction–response pairs: 104K detailed justifications with fine-grained ratings generated through automated scoring and CoT aggregation, and 216K question–answer pairs expanded by an LLM in both what-and-how and yes-or-no formats.
• Figure 4: Illustration of the progressive tuning. In the first stage, the model is trained on coarsely annotated data to acquire an initial sense of quality. In the second stage, it is further trained on higher-quality and more diverse data, enhancing both its scoring capability and its ability to provide justifications.