Scaling-up Perceptual Video Quality Assessment

TL;DR

This work tackles the scarcity of labeled data in perceptual video quality assessment by introducing OmniVQA, a human-in-the-loop MIDB framework that scales VQA data through a three-branch pipeline (technical, in-context, aesthetic). It delivers OmniVQA-Chat-400K, the largest VQA MIDB to date, and OmniVQA-MOS-20K for quality rating, complemented by a complementary training strategy that leverages both tasks to train specialized LMMs for rating and understanding. A dedicated OmniVQA-FG-Benchmark enables fine-grained spatiotemporal quality evaluation, while synthetic distortions and bounding-box semantic descriptions enrich the in-context data. Across experiments, the models achieve state-of-the-art performance on both rating and understanding tasks, with data scaling and complementary training demonstrating robust gains and practical implications for scalable, high-quality perceptual VQA systems.

Abstract

The data scaling law has been shown to significantly enhance the performance of large multi-modal models (LMMs) across various downstream tasks. However, in the domain of perceptual video quality assessment (VQA), the potential of scaling law remains unprecedented due to the scarcity of labeled resources and the insufficient scale of datasets. To address this, we propose \textbf{OmniVQA}, an efficient framework designed to efficiently build high-quality, human-in-the-loop VQA multi-modal instruction databases (MIDBs). We then scale up to create \textbf{OmniVQA-Chat-400K}, the largest MIDB in the VQA field concurrently. Our focus is on the technical and aesthetic quality dimensions, with abundant in-context instruction data to provide fine-grained VQA knowledge. Additionally, we have built the \textbf{OmniVQA-MOS-20K} dataset to enhance the model's quantitative quality rating capabilities. We then introduce a \textbf{complementary} training strategy that effectively leverages the knowledge from datasets for quality understanding and quality rating tasks. Furthermore, we propose the \textbf{OmniVQA-FG (fine-grain)-Benchmark} to evaluate the fine-grained performance of the models. Our results demonstrate that our models achieve state-of-the-art performance in both quality understanding and rating tasks.

Figures (28)

• Figure 1: OmniVQA overview. On the left side, it summarizes the existing MIDB construction paradigms in the visual quality assessment field. The main drawbacks of these methods lie in their excessive reliance on manual annotations or existing datasets, as well as the lack of comprehensive and diverse annotation dimensions. To address this, we have developed the OmniVQA datasets and models. On the right side, it shows examples of the OmniVQA model's application and the radar charts of its superior performance on two benchmarks.
• Figure 2: Data construction pipeline of OmniVQA-Chat-400K.
• Figure 3: Statistical information of OmniVQA-Chat-400K and OmniVQA-MOS-20K .
• Figure 4: Statistical information of the OmniVQA-FG-Bench.
• Figure 5: Examples of questions and correct answers of the human-annotated part of the OmniVQA-FG-Benchmark.