MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations

TL;DR

This work presents MedQ-Deg, a comprehensive benchmark for evaluating medical MLLMs under image quality degradations, and introduces Calibration Shift metric, which quantifies the gap between a model's perceived confidence and actual performance to assess metacognitive reliability under degradation.

Abstract

Despite impressive performance on standard benchmarks, multimodal large language models (MLLMs) face critical challenges in real-world clinical environments where medical images inevitably suffer various quality degradations. Existing benchmarks exhibit two key limitations: (1) absence of large-scale, multidimensional assessment across medical image quality gradients and (2) no systematic confidence calibration analysis. To address these gaps, we present MedQ-Deg, a comprehensive benchmark for evaluating medical MLLMs under image quality degradations. MedQ-Deg provides multi-dimensional evaluation spanning 18 distinct degradation types, 30 fine-grained capability dimensions, and 7 imaging modalities, with 24,894 question-answer pairs. Each degradation is implemented at 3 severity degrees, calibrated by expert radiologists. We further introduce Calibration Shift metric, which quantifies the gap between a model's perceived confidence and actual performance to assess metacognitive reliability under degradation. Our comprehensive evaluation of 40 mainstream MLLMs reveals several critical findings: (1) overall model performance degrades systematically as degradation severity increases, (2) models universally exhibit the AI Dunning-Kruger Effect, maintaining inappropriately high confidence despite severe accuracy collapse, and (3) models display markedly differentiated behavioral patterns across capability dimensions, imaging modalities, and degradation types. We hope MedQ-Deg drives progress toward medical MLLMs that are robust and trustworthy in real clinical practice.

Figures (11)

• Figure 1: Illustration of AI Dunning-Kruger Effect. An MLLM correctly identifies a liver lesion in the clean CT image but yields erroneous predictions with similarly high confidence when the image is corrupted while the lesion is still visible.
• Figure 2: Overview of the MedQ-Deg benchmark framework. Two orthogonal hierarchies structure the evaluation: a capability hierarchy (left) decomposing clinical competence into 30 fine-grained skills across 6 tasks, and a degradation hierarchy (right) covering 18 degradation types across 7 modalities. The data pipeline (middle) applies each degradation at three expert-calibrated severity degrees (L0--L2).
• Figure 3: Model performance across medical image quality degradation. Accuracy of 40 MLLMs evaluated at three severity degrees (L0--L2). X-axis labels are colored by model category: blue for open-source general models, brown for commercial MLLMs, and purple for medical-specialized models.
• Figure 4: Comprehensive analysis of model calibration shift across severity degrees. These panels quantify the widening gap between actual accuracy and model certainty under increasing image degradation. As degradation scales from L0 to L2, the stability of perceived performance against collapsing accuracy reveals a fundamental failure in metacognitive awareness across MLLMs.
• Figure 5: Simulated vs. Real-World Evaluation Consistency.Left: t-SNE projection of BiomedCLIP features from simulated and real degraded images. Real and simulated images co-locate within each modality cluster, confirming distribution alignment. Right: Bump chart tracing per-model ranks across Simulated-Mild, Simulated-Severe, Real-World-Mild, and Real-World-Severe conditions.