UNQA: Unified No-Reference Quality Assessment for Audio, Image, Video, and Audio-Visual Content

TL;DR

This paper introduces UNQA, a Unified No-reference Quality Assessment model capable of predicting perceptual quality for audio, image, video, and audio-visual content with a single set of parameters. It achieves this through three modality-specific feature extraction streams (spatial, motion, and audio) feeding four modality-specific regression heads, enabling both cross-modal and per-modality quality scores. A novel three-step multi-modality training strategy aligns perceptual scales across multiple QA databases by leveraging ranking information, yielding robust generalization and reduced model maintenance compared to modality-specific models. Experimental results across 12 QA databases demonstrate state-of-the-art performance and improved efficiency, with strong cross-database generalization and interpretable attention maps supporting its practical applicability in real-world multimedia processing and edge deployment.

Abstract

As multimedia data flourishes on the Internet, quality assessment (QA) of multimedia data becomes paramount for digital media applications. Since multimedia data includes multiple modalities including audio, image, video, and audio-visual (A/V) content, researchers have developed a range of QA methods to evaluate the quality of different modality data. While they exclusively focus on addressing the single modality QA issues, a unified QA model that can handle diverse media across multiple modalities is still missing, whereas the latter can better resemble human perception behaviour and also have a wider range of applications. In this paper, we propose the Unified No-reference Quality Assessment model (UNQA) for audio, image, video, and A/V content, which tries to train a single QA model across different media modalities. To tackle the issue of inconsistent quality scales among different QA databases, we develop a multi-modality strategy to jointly train UNQA on multiple QA databases. Based on the input modality, UNQA selectively extracts the spatial features, motion features, and audio features, and calculates a final quality score via the four corresponding modality regression modules. Compared with existing QA methods, UNQA has two advantages: 1) the multi-modality training strategy makes the QA model learn more general and robust quality-aware feature representation as evidenced by the superior performance of UNQA compared to state-of-the-art QA methods. 2) UNQA reduces the number of models required to assess multimedia data across different modalities. and is friendly to deploy to practical applications.

Figures (10)

• Figure 1: Most of previous QA models were trained on one QA database to measure the perceptual quality of a single modality. In this paper, we propose the first unified QA model that is jointly trained on multiple QA databases to predict qualities for inputs of different modalities.
• Figure 2: Images with approximately the same linearly rescaled MOS exhibit different perceptual quality. These images are sampled from (a) BID ciancio2010no, (b) CLIVE ghadiyaram2015massive, (c) KonIQ-10k hosu2020koniq (d) SPAQ fang2020perceptual. It is not hard to observe that the image (c) has the better quality than the other three.
• Figure 3: The overall architecture of the proposed model - Unified No-Reference Quality Assessment Model (UNQA). Our model is able to predict quality scores for different modalities: audio, image, video, and A/V. The model consists of three feature extraction modules and four modality-specific regression modules with minimal modality-specific changes.
• Figure 4: The architecture of the spatial feature extraction module. Assuming that the backbone model consists of $4$ stages, we extract feature maps from $3$ stages and feed them into MHSA to generate the spatial feature.
• Figure 5: The architecture of the audio feature extraction module. It consists of a framewise module, a time-dependency module, and an attention-pooling module.