Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMP-2K: A Benchmark Multi-Labeled Macro Photography Image Quality Assessment Database

Jiashuo Chang, Zhengyi Li, Jianxun Lou, Zhen Qiu, Hanhe Lin

TL;DR

The authors address the lack of benchmark MPIQA data by building MMP-2k, a 2,000-image macro photography quality dataset with MOS ratings and a rich distortion-quality report. They curate diverse content via a two-stage sampling pipeline from 15,700 images across three public sites, validated through pilot and main subjective studies with reliability checks. A detailed distortion annotation protocol accompanies the MOS measurements, enabling multi-label analysis of quality factors in MP images. Baseline evaluations show existing generic BIQA methods underperform on MP data, highlighting the need for MP-specific quality assessment approaches and positioning MMP-2k as a standard benchmark for development.

Abstract

Macro photography (MP) is a specialized field of photography that captures objects at an extremely close range, revealing tiny details. Although an accurate macro photography image quality assessment (MPIQA) metric can benefit macro photograph capturing, which is vital in some domains such as scientific research and medical applications, the lack of MPIQA data limits the development of MPIQA metrics. To address this limitation, we conducted a large-scale MPIQA study. Specifically, to ensure diversity both in content and quality, we sampled 2,000 MP images from 15,700 MP images, collected from three public image websites. For each MP image, 17 (out of 21 after outlier removal) quality ratings and a detailed quality report of distortion magnitudes, types, and positions are gathered by a lab study. The images, quality ratings, and quality reports form our novel multi-labeled MPIQA database, MMP-2k. Experimental results showed that the state-of-the-art generic IQA metrics underperform on MP images. The database and supplementary materials are available at https://github.com/Future-IQA/MMP-2k.

MMP-2K: A Benchmark Multi-Labeled Macro Photography Image Quality Assessment Database

TL;DR

The authors address the lack of benchmark MPIQA data by building MMP-2k, a 2,000-image macro photography quality dataset with MOS ratings and a rich distortion-quality report. They curate diverse content via a two-stage sampling pipeline from 15,700 images across three public sites, validated through pilot and main subjective studies with reliability checks. A detailed distortion annotation protocol accompanies the MOS measurements, enabling multi-label analysis of quality factors in MP images. Baseline evaluations show existing generic BIQA methods underperform on MP data, highlighting the need for MP-specific quality assessment approaches and positioning MMP-2k as a standard benchmark for development.

Abstract

Macro photography (MP) is a specialized field of photography that captures objects at an extremely close range, revealing tiny details. Although an accurate macro photography image quality assessment (MPIQA) metric can benefit macro photograph capturing, which is vital in some domains such as scientific research and medical applications, the lack of MPIQA data limits the development of MPIQA metrics. To address this limitation, we conducted a large-scale MPIQA study. Specifically, to ensure diversity both in content and quality, we sampled 2,000 MP images from 15,700 MP images, collected from three public image websites. For each MP image, 17 (out of 21 after outlier removal) quality ratings and a detailed quality report of distortion magnitudes, types, and positions are gathered by a lab study. The images, quality ratings, and quality reports form our novel multi-labeled MPIQA database, MMP-2k. Experimental results showed that the state-of-the-art generic IQA metrics underperform on MP images. The database and supplementary materials are available at https://github.com/Future-IQA/MMP-2k.

Paper Structure

This paper contains 12 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. MP image sampling
  3. Source images collection
  4. Diversity sampling
  5. Subjective MPIQA
  6. Pilot Study
  7. Main Study
  8. Result and Analysis
  9. Outlier Removal and MOS Generation
  10. Quality Report Generation
  11. Generic BIQA Evaluation
  12. Conclusion

Figures (5)

  • Figure 1: MP images in the MMP-2k database have two labels, a MOS (mean opinion score) obtained from 21 participants and a quality report describing overall quality, magnitudes, types, and positions of distortions annotated by two participants.
  • Figure 2: The flowchart of MP image sampling. 15,700 images with a tag of "macro" or "macro photography" are acquired from three public image websites, from which 2,000 MP images are sampled considering quality and content diversity.
  • Figure 3: Our subjective MPIQA consists of a pilot study and a main study. In the pilot study, 11 participants were invited to rate the qualities of 72 randomly selected MP images, which were used as training and test questions. In the main study, 21 participants were invited to rate the quality of 2,060 sampled MP images, with training and test questions embedded to enhance and validate participants' reliability. Additionally, two participants annotated the distortion magnitudes, types, and positions.
  • Figure 4: Reliability analysis of subjective study. (a) Accuracy of the participants on the 60 test questions. (b) PLCC correlation between ratings of each participant and MOS.
  • Figure 5: MOS distribution of the MMP-2k database.