Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Q-Bench-Portrait: Benchmarking Multimodal Large Language Models on Portrait Image Quality Perception

Sijing Wu, Yunhao Li, Zicheng Zhang, Qi Jia, Xinyue Li, Huiyu Duan, Xiongkuo Min, Guangtao Zhai

TL;DR

The paper addresses the lack of portrait-specific, low-level visual quality benchmarks for multimodal large language models (MLLMs). It introduces Q-Bench-Portrait, a holistic dataset of 2,765 image–question–answer triplets drawn from 17 portrait sources, organized around three quality dimensions (technical distortions, AIGC-specific distortions, aesthetics) and four question types (single-choice, multiple-choice, true/false, open-ended) at global and local scopes. By evaluating 25 MLLMs (open- and closed-source), the study shows that while some open-source models approach closed-model performance, all models struggle with fine-grained, portrait-specific quality perception, especially for AI-generated and computer-graphics portraits and for local questions. The benchmark, constructed with GPT-assisted human-in-the-loop methods and Gemini-generated captions, provides a rigorous framework to advance portrait-aware MLLMs and guides future improvements in perceptual quality assessment for human-centric imagery.

Abstract

Recent advances in multimodal large language models (MLLMs) have demonstrated impressive performance on existing low-level vision benchmarks, which primarily focus on generic images. However, their capabilities to perceive and assess portrait images, a domain characterized by distinct structural and perceptual properties, remain largely underexplored. To this end, we introduce Q-Bench-Portrait, the first holistic benchmark specifically designed for portrait image quality perception, comprising 2,765 image-question-answer triplets and featuring (1) diverse portrait image sources, including natural, synthetic distortion, AI-generated, artistic, and computer graphics images; (2) comprehensive quality dimensions, covering technical distortions, AIGC-specific distortions, and aesthetics; and (3) a range of question formats, including single-choice, multiple-choice, true/false, and open-ended questions, at both global and local levels. Based on Q-Bench-Portrait, we evaluate 20 open-source and 5 closed-source MLLMs, revealing that although current models demonstrate some competence in portrait image perception, their performance remains limited and imprecise, with a clear gap relative to human judgments. We hope that the proposed benchmark will foster further research into enhancing the portrait image perception capabilities of both general-purpose and domain-specific MLLMs.

Q-Bench-Portrait: Benchmarking Multimodal Large Language Models on Portrait Image Quality Perception

TL;DR

The paper addresses the lack of portrait-specific, low-level visual quality benchmarks for multimodal large language models (MLLMs). It introduces Q-Bench-Portrait, a holistic dataset of 2,765 image–question–answer triplets drawn from 17 portrait sources, organized around three quality dimensions (technical distortions, AIGC-specific distortions, aesthetics) and four question types (single-choice, multiple-choice, true/false, open-ended) at global and local scopes. By evaluating 25 MLLMs (open- and closed-source), the study shows that while some open-source models approach closed-model performance, all models struggle with fine-grained, portrait-specific quality perception, especially for AI-generated and computer-graphics portraits and for local questions. The benchmark, constructed with GPT-assisted human-in-the-loop methods and Gemini-generated captions, provides a rigorous framework to advance portrait-aware MLLMs and guides future improvements in perceptual quality assessment for human-centric imagery.

Abstract

Recent advances in multimodal large language models (MLLMs) have demonstrated impressive performance on existing low-level vision benchmarks, which primarily focus on generic images. However, their capabilities to perceive and assess portrait images, a domain characterized by distinct structural and perceptual properties, remain largely underexplored. To this end, we introduce Q-Bench-Portrait, the first holistic benchmark specifically designed for portrait image quality perception, comprising 2,765 image-question-answer triplets and featuring (1) diverse portrait image sources, including natural, synthetic distortion, AI-generated, artistic, and computer graphics images; (2) comprehensive quality dimensions, covering technical distortions, AIGC-specific distortions, and aesthetics; and (3) a range of question formats, including single-choice, multiple-choice, true/false, and open-ended questions, at both global and local levels. Based on Q-Bench-Portrait, we evaluate 20 open-source and 5 closed-source MLLMs, revealing that although current models demonstrate some competence in portrait image perception, their performance remains limited and imprecise, with a clear gap relative to human judgments. We hope that the proposed benchmark will foster further research into enhancing the portrait image perception capabilities of both general-purpose and domain-specific MLLMs.
Paper Structure (14 sections, 4 figures, 6 tables)

This paper contains 14 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Model
  4. Multimodal Benchmark
  5. Visual Quality Assessment
  6. Q-Bench-Portrait
  7. Portrait Image Collection
  8. Quality Dimension
  9. Question Design
  10. Benchmark Construction & Quality Control
  11. Experiments
  12. Experimental Setting
  13. Main Results
  14. Conclusion

Figures (4)

  • Figure 1: Examples from Q-Bench-Portrait. We present images and their corresponding question–answer pairs across different image categories. Notably, for each category, the images are associated with questions covering at least one of the three quality dimensions (i.e., technical distortions, AIGC-specific distortions, and aesthetics) and all four question types (i.e., SC, MC, TF, and OE).
  • Figure 2: Distributions of (a) image sources, (b) quality dimensions, (c) question types, and (d) question scopes.
  • Figure 3: Illustration of the GUI used to review and refine questions and answers in Q-Bench-Portrait.
  • Figure 4: Radar chart of benchmark results. (a) Comparison of overall performance across all MLLMs. (b) Comparison of five representative MLLMs across different subsets, including image types, quality dimensions, and question types.