Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution

TL;DR

This paper tackles the challenge of regressing accurate image quality scores with multi-modal LLMs by addressing the distribution mismatch between continuous quality scores and discrete token outputs. It introduces a distribution-based soft label that discretizes the score distribution across five levels, coupled with a Thurstone-model fidelity loss to enable robust multi-dataset co-training. The proposed DeQA-Score framework combines a vision encoder, a visual abstractor, and an LLM to predict both score means and distributions, achieving superior score regression and distribution alignment across diverse IQA benchmarks. The method demonstrates strong cross-dataset generalization and real-time inference potential, offering practical impact for scalable, distribution-aware IQA with MLLMs.

Abstract

With the rapid advancement of Multi-modal Large Language Models (MLLMs), MLLM-based Image Quality Assessment (IQA) methods have shown promising performance in linguistic quality description. However, current methods still fall short in accurately scoring image quality. In this work, we aim to leverage MLLMs to regress accurate quality scores. A key challenge is that the quality score is inherently continuous, typically modeled as a Gaussian distribution, whereas MLLMs generate discrete token outputs. This mismatch necessitates score discretization. Previous approaches discretize the mean score into a one-hot label, resulting in information loss and failing to capture inter-image relationships. We propose a distribution-based approach that discretizes the score distribution into a soft label. This method preserves the characteristics of the score distribution, achieving high accuracy and maintaining inter-image relationships. Moreover, to address dataset variation, where different IQA datasets exhibit various distributions, we introduce a fidelity loss based on Thurstone's model. This loss captures intra-dataset relationships, facilitating co-training across multiple IQA datasets. With these designs, we develop the distribution-based Depicted image Quality Assessment model for Score regression (DeQA-Score). Experiments across multiple benchmarks show that DeQA-Score stably outperforms baselines in score regression. Also, DeQA-Score can predict the score distribution that closely aligns with human annotations. Codes and model weights have been released in https://depictqa.github.io/deqa-score/.

Figures (11)

• Figure 1: Illustration of our distribution-based soft label. Image A is overexposed, while Image B and C are well-exposed with better and similar quality. "exce." means "excellent". To train an MLLM as a quality scorer, continuous scores need to be discretized to discrete level tokens as the training label. Q-Align qalign discretizes human-labeled mean score to a one-hot label, causing information loss. We discretize the score distribution, which is approximated as a Gaussian distribution, to obtain a more accurate soft label. Our soft label maintains better inter-image relationships that the Image B is better than A (${\rm A < B}$), and Image B and C share similar quality (${\rm B \approx C}$), while the one-hot label can not.
• Figure 2: (a) MLLMs need discrete level tokens for training. Thus, continuous scores must be converted into discrete level tokens as training labels. (b) Comparison between one-hot label qalign and our soft label. One-hot label uniformly divides the score range into 5 intervals to discretize the mean score, i.e., the expectation of estimated score distribution. One-hot label assumes that different levels are identically independent, which is inaccurate. In contrast, we discretize the estimated Gaussian distribution of the score. Our soft label better preserves the relationships between levels, e.g., "fair" is closer to "good" than "excellent".
• Figure 3: Illustration of the soft label adjustment.
• Figure 4: Framework of our DeQA-Score trained with soft label. For the "$<$level$>$" token, the KL divergence loss is calculated between predicted probabilities and our soft label. For other tokens, common cross-entropy loss for next token prediction is calculated.
• Figure 5: Fidelity loss to maintain intra-dataset relationships. Consider two images, Image A and B, sampled from the same dataset. Using our distribution-based soft label, DeQA-Score predicts the expectation, $\mu^{pred}$, and variance, $(\sigma^{pred})^2$, of each image’s score distribution, with which we can predict the probability that Image A is better than Image B, $P(A>B)$. This predicted probability is then used to calculate the fidelity loss with the ground-truth probability. Fidelity loss preserves intra-dataset relationships, facilitating the co-training across multiple IQA datasets.