Image Quality Assessment: From Human to Machine Preference

TL;DR

The paper introduces Image Quality Assessment for Machine Vision by formalizing machine-centric preferences across downstream tasks and constructing the Machine Preference Database (MPD) with 2.25 million annotations on 30,000 reference/distorted image pairs spanning NSIs, SCIs, and AIGIs. It demonstrates that existing human-centric IQA metrics poorly predict machine preferences, highlighting a fundamental gap between human and machine perception. Through comprehensive database construction, validation, and cross-dataset experiments, the work shows both the potential and limitations of current metrics and advocates for task-aware, machine-focused IQA development. MPD provides a foundation for evaluating and guiding IQA methods tailored to machine vision in a rapidly evolving landscape of LMMs and CV models.

Abstract

Image Quality Assessment (IQA) based on human subjective preferences has undergone extensive research in the past decades. However, with the development of communication protocols, the visual data consumption volume of machines has gradually surpassed that of humans. For machines, the preference depends on downstream tasks such as segmentation and detection, rather than visual appeal. Considering the huge gap between human and machine visual systems, this paper proposes the topic: Image Quality Assessment for Machine Vision for the first time. Specifically, we (1) defined the subjective preferences of machines, including downstream tasks, test models, and evaluation metrics; (2) established the Machine Preference Database (MPD), which contains 2.25M fine-grained annotations and 30k reference/distorted image pair instances; (3) verified the performance of mainstream IQA algorithms on MPD. Experiments show that current IQA metrics are human-centric and cannot accurately characterize machine preferences. We sincerely hope that MPD can promote the evolution of IQA from human to machine preferences. Project page is on: https://github.com/lcysyzxdxc/MPD.

Figures (12)

• Figure 1: The significant gap between the well-explored Human Vision System (HVS) and the emerging Machine Vision System (MVS). Their different perception mechanisms leading to images that are subjectively satisfactory to humans not being applicable to the machines downstream tasks such as detection and question answering, vice versa.
• Figure 2: Overview of Machine Preference Database (MPD), with 30k large-scale reference/distorted image pairs, meticulously annotated with 2.25M fine-grained preference score from the mean opinion of 15 mainstream LMMs and 15 specific CV models. It provides a multi-dimensional fidelity evaluation of various downstream machine vision tasks.
• Figure 3: Correlation between the general preference score and the score in seven different downstream tasks.
• Figure 4: MOS score of MPD, visualized in 30 corruption subsets. Different color denotes corruption strength Level 1-Level 2-Level 3-Level 4-Level 5. Results show the sensitivity of the machines to each corruption varied significantly.
• Figure 5: Annotations from different LMMs subjects in MPD under 5 corruption strengths, including 3 content types and 4 tasks.