Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

meval: A Statistical Toolbox for Fine-Grained Model Performance Analysis

Dishantkumar Sutariya, Eike Petersen

TL;DR

The paper presents meval, a Python toolbox for rigorous, fine-grained assessment of model performance across intersectional subgroups in medical imaging. It covers metric design (including base-rate independent metrics like pAUPRG and DRMSCE), confidence intervals, permutation-based hypothesis testing with Holm correction, and scalable visualization, enabling robust detection of subgroup disparities. Two case studies (ISIC2020 and MIMIC-CXR) illustrate how disparities can arise in specific subgroups even when aggregate metrics like AUROC are similar, underscoring the need for intersectional auditing. The toolbox outputs interactive HTML reports and raw results to standardize and facilitate widespread use of rigorous subgroup performance analyses in clinical imaging contexts.

Abstract

Analyzing machine learning model performance stratified by patient and recording properties is becoming the accepted norm and often yields crucial insights about important model failure modes. Performing such analyses in a statistically rigorous manner is non-trivial, however. Appropriate performance metrics must be selected that allow for valid comparisons between groups of different sample sizes and base rates; metric uncertainty must be determined and multiple comparisons be corrected for, in order to assess whether any observed differences may be purely due to chance; and in the case of intersectional analyses, mechanisms must be implemented to find the most `interesting' subgroups within combinatorially many subgroup combinations. We here present a statistical toolbox that addresses these challenges and enables practitioners to easily yet rigorously assess their models for potential subgroup performance disparities. While broadly applicable, the toolbox is specifically designed for medical imaging applications. The analyses provided by the toolbox are illustrated in two case studies, one in skin lesion malignancy classification on the ISIC2020 dataset and one in chest X-ray-based disease classification on the MIMIC-CXR dataset.

meval: A Statistical Toolbox for Fine-Grained Model Performance Analysis

TL;DR

The paper presents meval, a Python toolbox for rigorous, fine-grained assessment of model performance across intersectional subgroups in medical imaging. It covers metric design (including base-rate independent metrics like pAUPRG and DRMSCE), confidence intervals, permutation-based hypothesis testing with Holm correction, and scalable visualization, enabling robust detection of subgroup disparities. Two case studies (ISIC2020 and MIMIC-CXR) illustrate how disparities can arise in specific subgroups even when aggregate metrics like AUROC are similar, underscoring the need for intersectional auditing. The toolbox outputs interactive HTML reports and raw results to standardize and facilitate widespread use of rigorous subgroup performance analyses in clinical imaging contexts.

Abstract

Analyzing machine learning model performance stratified by patient and recording properties is becoming the accepted norm and often yields crucial insights about important model failure modes. Performing such analyses in a statistically rigorous manner is non-trivial, however. Appropriate performance metrics must be selected that allow for valid comparisons between groups of different sample sizes and base rates; metric uncertainty must be determined and multiple comparisons be corrected for, in order to assess whether any observed differences may be purely due to chance; and in the case of intersectional analyses, mechanisms must be implemented to find the most `interesting' subgroups within combinatorially many subgroup combinations. We here present a statistical toolbox that addresses these challenges and enables practitioners to easily yet rigorously assess their models for potential subgroup performance disparities. While broadly applicable, the toolbox is specifically designed for medical imaging applications. The analyses provided by the toolbox are illustrated in two case studies, one in skin lesion malignancy classification on the ISIC2020 dataset and one in chest X-ray-based disease classification on the MIMIC-CXR dataset.

Paper Structure

This paper contains 14 sections, 2 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Metric choices.
  5. Intersectional analyses
  6. Confidence intervals
  7. Statistical hypothesis testing
  8. Implementation
  9. Case studies
  10. ISIC skin lesion malignancy classification
  11. MIMIC-CXR lung disease diagnosis
  12. Conclusion and outlook
  13. Acknowledgments.
  14. Disclosure of Interests.

Figures (3)

  • Figure 1: Exemplary default output of the toolbox for the ISIC2020 case study. Statistical tests performed for accuracy and AUROC. For thresholded metrics, the base rate was chosen as the decision threshold. H/N: head/neck, LE: lower extremity, UE: upper extremity, O/G: oral/genital, TO: torso. ns: not significant ($p > 0.01$), *: $p \leq 0.01$, **: $p \leq 0.001$. Dashed vertical lines indicate that CIs could not be obtained.
  • Figure 2: No-Finding Specificity (left) and AUROC (right), stratified by racial groups. For specificity, the threshold was chosen as in SeyyedKalantari2021 to maximize the geometric mean of sensitivity and specificity. ns: not significant ($p > 0.01$), *: $p \leq 0.01$, **: $p \leq 0.001$.
  • Figure 3: ROC curves and operating points for white (red) and black (purple) patients.