Fairness Evaluation of Risk Estimation Models for Lung Cancer Screening

TL;DR

This work assesses fairness in AI-driven lung cancer risk estimation for LDCT screening by applying the JustEFAB framework to three models (Sybil, Venkadesh21, PanCan2b) trained on NLST data. It employs a two-stage analysis (subgroup performance and fairness assessment) across demographic factors, reporting notable AUROC and operating-point disparities, particularly by sex and race, that persist after accounting for known confounders. The findings highlight underrepresented groups and socioeconomic factors as sources of unfair biases, emphasizing the need for more diverse data and fairness-guided deployment in screening. The study advocates for ongoing fairness monitoring, dataset diversification, and algorithmic mitigation strategies to ensure equitable lung cancer screening outcomes.

Abstract

Lung cancer is the leading cause of cancer-related mortality in adults worldwide. Screening high-risk individuals with annual low-dose CT (LDCT) can support earlier detection and reduce deaths, but widespread implementation may strain the already limited radiology workforce. AI models have shown potential in estimating lung cancer risk from LDCT scans. However, high-risk populations for lung cancer are diverse, and these models' performance across demographic groups remains an open question. In this study, we drew on the considerations on confounding factors and ethically significant biases outlined in the JustEFAB framework to evaluate potential performance disparities and fairness in two deep learning risk estimation models for lung cancer screening: the Sybil lung cancer risk model and the Venkadesh21 nodule risk estimator. We also examined disparities in the PanCan2b logistic regression model recommended in the British Thoracic Society nodule management guideline. Both deep learning models were trained on data from the US-based National Lung Screening Trial (NLST), and assessed on a held-out NLST validation set. We evaluated AUROC, sensitivity, and specificity across demographic subgroups, and explored potential confounding from clinical risk factors. We observed a statistically significant AUROC difference in Sybil's performance between women (0.88, 95% CI: 0.86, 0.90) and men (0.81, 95% CI: 0.78, 0.84, p < .001). At 90% specificity, Venkadesh21 showed lower sensitivity for Black (0.39, 95% CI: 0.23, 0.59) than White participants (0.69, 95% CI: 0.65, 0.73). These differences were not explained by available clinical confounders and thus may be classified as unfair biases according to JustEFAB. Our findings highlight the importance of improving and monitoring model performance across underrepresented subgroups, and further research on algorithmic fairness, in lung cancer screening.

Figures (7)

• Figure 1: Flowchart of subgroup performance analysis (left) and fairness assessment (right) for a single model on a single demographic characteristic. Color coding indicates the decision paths to follow to reach a result (ex. disparity found).
• Figure 2: ROC curves (with 95% CIs) for Sybil (Year 1) for men and women on NLST (n=5911 scans).
• Figure 3: Selected sensitivity and specificity of models on specific thresholds on NLST (n=5911 scans).
• Figure 4: ROC curves (with 95% CIs) for Sybil (Year 1) between men and women when isolating for a self-reported work history of working in a dangerous occupation for the lungs with or without a mask.
• Figure 5: ROC curves (with 95% CIs) for Venkadesh21 between racial groups when isolating for a hypertension diagnosis.