A Comprehensive Study of Multimodal Large Language Models for Image Quality Assessment

TL;DR

This work systematically investigates prompting Multimodal Large Language Models (MLLMs) for Image Quality Assessment (IQA) by pairing psychophysics-inspired testing methods with NLP prompting strategies, and introduces a difficult-sample selection procedure using expert IQA models as proxies. It evaluates three open-source and one closed-source MLLM across full-reference and no-reference IQA tasks over multiple visual attributes, revealing that GPT-4V generally provides the strongest alignment with human perception, but struggles with fine-grained color differences and multi-image comparisons. Open-source MLLMs require model-specific prompting to approach IQA performance, and chain-of-thought prompting offers notable gains for GPT-4V, suggesting that IQA benefits from integrating perceptual analysis into broader reasoning tasks. The study highlights the need for prompt optimization and cautious interpretation of open-model capabilities, while proposing a practical sampling framework to efficiently stress-test MLLMs on IQA benchmarks. These findings inform future directions in using MLLMs for interpretable, text-driven IQA and model evaluation.

Abstract

While Multimodal Large Language Models (MLLMs) have experienced significant advancement in visual understanding and reasoning, their potential to serve as powerful, flexible, interpretable, and text-driven models for Image Quality Assessment (IQA) remains largely unexplored. In this paper, we conduct a comprehensive and systematic study of prompting MLLMs for IQA. We first investigate nine prompting systems for MLLMs as the combinations of three standardized testing procedures in psychophysics (i.e., the single-stimulus, double-stimulus, and multiple-stimulus methods) and three popular prompting strategies in natural language processing (i.e., the standard, in-context, and chain-of-thought prompting). We then present a difficult sample selection procedure, taking into account sample diversity and uncertainty, to further challenge MLLMs equipped with the respective optimal prompting systems. We assess three open-source and one closed-source MLLMs on several visual attributes of image quality (e.g., structural and textural distortions, geometric transformations, and color differences) in both full-reference and no-reference scenarios. Experimental results show that only the closed-source GPT-4V provides a reasonable account for human perception of image quality, but is weak at discriminating fine-grained quality variations (e.g., color differences) and at comparing visual quality of multiple images, tasks humans can perform effortlessly.

Figures (6)

• Figure 1: Illustration of visual attributes of image quality in our experiments.
• Figure 2: Three standardized psychophysical testing procedures for IQA. (a) Single-stimulus method. (b) Double-stimulus method. (c) Multiple-stimulus method.
• Figure 3: Instantiations of systematic prompting strategies for GPT-4V in the NR scenario. (a) Standard prompting. (b) Chain-of-thought prompting. (c) In-context prompting. See complete FR and NR text prompts in the supplementary material.
• Figure 4: Comparison between difficult sample selection with and without variance normalization under the same level of sample diversity.
• Figure 5: Behaviors of different MLLMs in recognizing objects from multiple images.