DSMix: Distortion-Induced Sensitivity Map Based Pre-training for No-Reference Image Quality Assessment

TL;DR

DSMix tackles the data-scarcity challenge in No-Reference Image Quality Assessment by introducing a distortion-aware data augmentation built on distortion-induced sensitivity maps (DSMs). It combines DSM-based pre-training with a semantic knowledge distillation framework from an ImageNet teacher, then applies a lightweight linear probing stage to map DSM-informed features to quality scores. The approach yields state-of-the-art or near-state-of-the-art NR-IQA performance across seven datasets, while maintaining low training cost (single-GPU) and strong generalization, including effective cross-dataset results. The method provides practical impact by enabling robust IQA pre-training with synthetic distortions and semantic cues, enhancing applicability to authentic distortions in real-world scenarios.

Abstract

Image quality assessment (IQA) has long been a fundamental challenge in image understanding. In recent years, deep learning-based IQA methods have shown promising performance. However, the lack of large amounts of labeled data in the IQA field has hindered further advancements in these methods. This paper introduces DSMix, a novel data augmentation technique specifically designed for IQA tasks, aiming to overcome this limitation. DSMix leverages the distortion-induced sensitivity map (DSM) of an image as prior knowledge. It applies cut and mix operations to diverse categories of synthetic distorted images, assigning confidence scores to class labels based on the aforementioned prior knowledge. In the pre-training phase using DSMix-augmented data, knowledge distillation is employed to enhance the model's ability to extract semantic features. Experimental results on both synthetic and authentic IQA datasets demonstrate the significant predictive and generalization performance achieved by DSMix, without requiring fine-tuning of the full model. Code is available at \url{https://github.com/I2-Multimedia-Lab/DSMix}.

Figures (6)

• Figure 1: An overview of DSMix method. Dist. A and Dist. B denote distorted images with corresponding distortion type labels $\mathcal{Y}_A$ and $\mathcal{Y}_B$, respectively. By performing cut and mix operations on these two types of distorted images, the mixed image $\mathcal{I}_{Mix}$ is generated with a corresponding label $\mathcal{Y}_{Mix}$. The value of $\lambda$ is obtained by calculating the area ratio of $\mathcal{DSM}_A$ (blue area) and $\mathcal{DSM}_B$ (orange area) with respect to the entire DSM.
• Figure 2: An overview of our proposed NR-IQA method based on DSMix. (a) The generation of DSM. (b) Self-supervised pre-training based on DSMix. (c) Linear probing on the specific IQA dataset.
• Figure 3: Overview of GT DSM generation process.
• Figure 4: Overview of semantic knowledge distillation, where the flame and snowflake icons denote parameters frozen and tuned, respectively. Enc-S denotes QE, and Enc-T denotes teacher model.
• Figure 5: Comparison for 2D t-SNE visualization of learned representations between CONTRIQUE ("-C") and DSMix ("-D"). Zooming in for a better view.