Subgroup Performance Analysis in Hidden Stratifications

TL;DR

This work tackles the challenge of hidden stratifications causing performance disparities in medical imaging models by introducing subgroup discovery as a tool for performance monitoring. The authors implement a DOMINO-based approach that clusters task-agnostic image representations with a balancing parameter $ abla$, enabling discovery of cohesive subgroups without requiring ground-truth subgroup labels. Through synthetic artifacts and real-world datasets (CheXpert-Plus and SLICE-3D), they show that discovered subgroups reveal larger performance gaps than metadata-based subgroups while maintaining cohesion, and that these gaps align with meaningful visual features rather than demographics. Their results justify using subgroup discovery as a practical, robust complement to traditional subgroup analysis for safer and more trustworthy AI deployment in medicine, with implications for both validation and bias mitigation. Key metrics include the performance gap $Δ(S)$ and average purity $AP(S)$, and the findings demonstrate that natural image feature extractors are often sufficient to expose important disparities, supporting broader applicability in clinical settings.

Abstract

Machine learning (ML) models may suffer from significant performance disparities between patient groups. Identifying such disparities by monitoring performance at a granular level is crucial for safely deploying ML to each patient. Traditional subgroup analysis based on metadata can expose performance disparities only if the available metadata (e.g., patient sex) sufficiently reflects the main reasons for performance variability, which is not common. Subgroup discovery techniques that identify cohesive subgroups based on learned feature representations appear as a potential solution: They could expose hidden stratifications and provide more granular subgroup performance reports. However, subgroup discovery is challenging to evaluate even as a standalone task, as ground truth stratification labels do not exist in real data. Subgroup discovery has thus neither been applied nor evaluated for the application of subgroup performance monitoring. Here, we apply subgroup discovery for performance monitoring in chest x-ray and skin lesion classification. We propose novel evaluation strategies and show that a simplified subgroup discovery method without access to classification labels or metadata can expose larger performance disparities than traditional metadata-based subgroup analysis. We provide the first compelling evidence that subgroup discovery can serve as an important tool for comprehensive performance validation and monitoring of trustworthy AI in medicine.

Figures (4)

• Figure 1: (a) Traditional subgroup analysis detects disparate patient outcomes, but it is limited to annotated metadata. (b) Subgroup discovery reveals hidden stratifications but lacks performance validation. (c) We bridge this gap by applying subgroup discovery for performance analysis in both (d) controlled synthetic settings and (e) real-world scenarios with unknown subgroups.
• Figure 2: Performance gap and purity of subgroups across different $\gamma$ and bias levels.
• Figure 3: Detailed subgroup accuracies for our synthetic scenario. Purer subgroups performances (darker dots) capture the true performance gap characterized by hidden subgroups, which are overlooked by traditional subgroup analysis with access to a single artifact (known subgroups), and by overall performance.
• Figure 4: (a,c): Performance gaps and metadata-based purity for different $\gamma$. (b,d): Histograms of subgroup performances for different subgroup divisions: In blue, subgroup discovery with different external CLIP models, averaged over different random seeds (gray transparent bars). In red, our baseline of subgroups defined by different metadata, averaged over their attributes (gray). Top (a,b) and bottom (c,d) rows show CheXpertPlus and SLICE-3D results, respectively.