Metric Hub: A metric library and practical selection workflow for use-case-driven data quality assessment in medical AI

TL;DR

The paper addresses the lack of systematic, use-case driven data quality assessment for medical AI by operationalizing the METRIC-framework into a metric library of 60 quantitative metrics, each with standardized metric cards and context-aware decision trees. It introduces the Metric Hub platform to host the library and demonstrates a practical workflow on the PTB-XL ECG dataset, selecting and evaluating metrics for fit-for-purpose data quality across original and perturbed subsets. The contributions include the harmonized metric library, a decision-tree based metric selection workflow, and a real-world demonstration, highlighting both the utility and current gaps such as thresholding and cross-dataset comparability. Together, these tools enable structured, task-specific evaluation of training and test data in medicine, supporting trustworthy AI development and regulatory alignment.

Abstract

Machine learning (ML) in medicine has transitioned from research to concrete applications aimed at supporting several medical purposes like therapy selection, monitoring and treatment. Acceptance and effective adoption by clinicians and patients, as well as regulatory approval, require evidence of trustworthiness. A major factor for the development of trustworthy AI is the quantification of data quality for AI model training and testing. We have recently proposed the METRIC-framework for systematically evaluating the suitability (fit-for-purpose) of data for medical ML for a given task. Here, we operationalize this theoretical framework by introducing a collection of data quality metrics - the metric library - for practically measuring data quality dimensions. For each metric, we provide a metric card with the most important information, including definition, applicability, examples, pitfalls and recommendations, to support the understanding and implementation of these metrics. Furthermore, we discuss strategies and provide decision trees for choosing an appropriate set of data quality metrics from the metric library given specific use cases. We demonstrate the impact of our approach exemplarily on the PTB-XL ECG-dataset. This is a first step to enable fit-for-purpose evaluation of training and test data in practice as the base for establishing trustworthy AI in medicine.

Figures (21)

• Figure 1: Recommended workflow for multi-dimensional data quality evaluation: Identify the data quality dimensions most relevant for risk and intended use; implement and parametrize metrics using the metric cards; follow dimension-specific decision trees to select applicable metrics.
• Figure 2: Overview of metrics in metric library. a: The number of metrics associated to the seven disjoint groups (Measurement process, consistency, representativeness, informativeness, timeliness, distribution metrics, correlation coefficients); b: The number of metrics per dimension in the METRIC-framework (with repeated counts); c: The distribution of metrics across different data modalities (tabular, image, time series, multimodal, text).
• Figure 3: Decision tree for the dimension "Accuracy" -- A practical path to fit-for-purpose metrics per data quality dimension. The decision tree guides the user from data quality dimension to a suitable metric/ set of metrics based on the use case. The green boxes "Correlation Coefficients" and "Distribution Metrics" guide to general subtrees (see Figure \ref{['fig:decisiontree-correlation']} and Figure \ref{['fig:decisiontree-distribution']}).
• Figure 4: Exemplary metric card for Entropy. This is one of 60 cards available on our related website Metrichub The cards summarize important information on each metric including definition, visualization and value range, example, applicability, pitfalls and recommendations.
• Figure 5: Revised version of the METRIC-framework for systematic ML data quality evaluation in medicine. The inner circle divides data quality into five clusters. These clusters contain a total of 26 data quality dimensions, which are shown on the outer circle. Related dimensions are summarised into groups at the borders.