VQA$^2$: Visual Question Answering for Video Quality Assessment

TL;DR

This work tackles the lack of holistic low-level video quality understanding in the era of large multi-modal models by introducing the VQA2 Instruction Dataset, a three-stage, instruction-tuning resource for video quality assessment. It proposes the VQA2 series models, which interleave visual and motion tokens to capture spatial-temporal quality details, including specialized VQA2-Scorers for scoring and the VQA2-Assistant for understanding. The results show state-of-the-art performance on video quality scoring and that the VQA2-Assistant can surpass GPT-4o on quality understanding, highlighting a versatile approach that unifies scoring and understanding in video quality tasks. This dataset and pipeline provide a foundation for deploying LMM-guided, low-level video quality assessment across encoding, transmission, and generation workflows, with open-source release planned for broader adoption.

Abstract

The advent and proliferation of large multi-modal models (LMMs) have introduced new paradigms to computer vision, transforming various tasks into a unified visual question answering framework. Video Quality Assessment (VQA), a classic field in low-level visual perception, focused initially on quantitative video quality scoring. However, driven by advances in LMMs, it is now progressing toward more holistic visual quality understanding tasks. Recent studies in the image domain have demonstrated that Visual Question Answering (VQA) can markedly enhance low-level visual quality evaluation. Nevertheless, related work has not been explored in the video domain, leaving substantial room for improvement. To address this gap, we introduce the VQA2 Instruction Dataset - the first visual question answering instruction dataset that focuses on video quality assessment. This dataset consists of 3 subsets and covers various video types, containing 157,755 instruction question-answer pairs. Then, leveraging this foundation, we present the VQA2 series models. The VQA2 series models interleave visual and motion tokens to enhance the perception of spatial-temporal quality details in videos. We conduct extensive experiments on video quality scoring and understanding tasks, and results demonstrate that the VQA2series models achieve excellent performance in both tasks. Notably, our final model, the VQA2-Assistant, exceeds the renowned GPT-4o in visual quality understanding tasks while maintaining strong competitiveness in quality scoring tasks. Our work provides a foundation and feasible approach for integrating low-level video quality assessment and understanding with LMMs.

Figures (12)

• Figure 1: The performance of the VQA2 series models on video quality scoring and video quality understanding tasks. To highlight performance differences, each dimension in all the radar charts is normalized using different ranges based on the specific data characteristics.
• Figure 2: Data construction pipelines of Stage-1 and Stage-2.
• Figure 3: Data construction pipeline of Stage-3. This subset is annotated by human experts and then refined and expanded through GPT.
• Figure 4: The model structure and training strategy. The model is based on the vanilla LLaVA-OneVision-Chat and SlowFast-R50. The training strategy (freeze or unfreeze) across the $3$ stages is separated by "$\rightarrow$". Specifically, "/" denotes the training strategy used during the "image data training/video data training" in Stage-1.
• Figure 5: Two different data-combining strategies.