Enhancing Blind Video Quality Assessment with Rich Quality-aware Features

TL;DR

This work tackles blind video quality assessment (BVQA) for social-media content by proposing RQ-VQA, a modular framework that fuses learnable spatial features from a target-domain base model with rich quality-aware representations sourced from off-the-shelf BIQA and BVQA models. The approach captures spatial, spatiotemporal, and temporal cues through components like Swin Transformer-based features, LIQE, Q-Align, FAST-VQA, and SlowFast, all concatenated and regressed by a lightweight MLP to predict video quality. Extensive experiments on KVQ, TaoLive, and LIVE-WC demonstrate state-of-the-art performance and strong generalization, with NTIRE 2024 Short-form UGC Challenge results ranking first. The method is efficient, modular, and adaptable, offering robust quality assessment for diverse social-media workflows and likely extensions to domain-specific content and graphics-aware scenarios.

Abstract

Blind video quality assessment (BVQA) is a highly challenging task due to the intrinsic complexity of video content and visual distortions, especially given the high popularity of social media videos, which originate from a wide range of sources, and are often processed by various compression and enhancement algorithms. While recent BVQA and blind image quality assessment (BIQA) studies have made remarkable progress, their models typically perform well on the datasets they were trained on but generalize poorly to unseen videos, making them less effective for accurately evaluating the perceptual quality of diverse social media videos. In this paper, we propose Rich Quality-aware features enabled Video Quality Assessment (RQ-VQA), a simple yet effective method to enhance BVQA by leveraging rich quality-aware features extracted from off-the-shelf BIQA and BVQA models. Our approach exploits the expertise of existing quality assessment models within their trained domains to improve generalization. Specifically, we design a multi-source feature framework that integrates:(1) Learnable spatial features} from a base model fine-tuned on the target VQA dataset to capture domain-specific quality cues; (2) Temporal motion features from the fast pathway of SlowFast pre-trained on action recognition datasets to model motion-related distortions; (3) Spatial quality-aware features from BIQA models trained on diverse IQA datasets to enhance frame-level distortion representation; and (4) Spatiotemporal quality-aware features from a BVQA model trained on large-scale VQA datasets to jointly encode spatial structure and temporal dynamics. These features are concatenated and fed into a multi-layer perceptron (MLP) to regress them into quality scores. Experimental results demonstrate that our model achieves state-of-the-art performance on three public social media VQA datasets.

Figures (3)

• Figure 1: The comparison of PGC videos, UGC videos, and the processed UGC videos.
• Figure 2: Overall architecture of the proposed RQ-VQA framework. Given an input video, key frames and chunks are first extracted for feature computation. The framework integrates multiple feature sources: (1) Learnable spatial features from Swin Transformer-B with an MHSA module, (2) Temporal features from the fast pathway of SlowFast, (3) Spatiotemporal quality-aware features from FAST-VQA, (4) Spatial quality-aware features from LIQE using cosine similarity between text and image embeddings, and (5) Spatial quality-aware features from Q-Align based on large multimodal model representations. All features are concatenated and fed into a two-layer MLP to regress the final quality score.
• Figure 3: Typical failure cases of RQ-VQA on social-media videos. In both examples, the predicted RQ-VQA scores are noticeably higher than the corresponding MOS values, indicating overestimation on text-overlay or synthetic-content scenes.