Opening the black box of deep learning: Validating the statistical association between explainable artificial intelligence (XAI) and clinical domain knowledge in fundus image-based glaucoma diagnosis
Han Yuan, Lican Kang, Yong Li
TL;DR
This work tackles the interpretability gap in deep learning for glaucoma diagnosis from fundus images by statistically validating that CAM-based explanations align with clinicians' anatomical knowledge of the optic cup, optic disk, and blood vessels. It trains four lightweight backbones—VGG-11, ResNet-18, DeiT-Tiny, and Swin-Tiny—on five public datasets and analyzes explanations generated by Grad-CAM, XGrad-CAM, Score-CAM, Eigen-CAM, and Layer-CAM, using paired-sample tests and correlations to relate focus-area anatomy to predictive performance. Key findings show consistent focus on anatomies beyond random expectation and a positive relationship between anatomy-focused attention and AUROC/AUPRC across internal and external datasets, with CNNs sometimes outperforming vision transformers in this middle-scale setting. By providing statistically grounded evidence of alignment between model reasoning and clinical knowledge and releasing reproducible resources on GitHub, the work advances trust and potential knowledge distillation in ophthalmology DL models.
Abstract
While deep learning has exhibited remarkable predictive capabilities in various medical image tasks, its inherent black-box nature has hindered its widespread implementation in real-world healthcare settings. Our objective is to unveil the decision-making processes of deep learning models in the context of glaucoma classification by employing several Class Activation Map (CAM) techniques to generate model focus regions and comparing them with clinical domain knowledge of the anatomical area (optic cup, optic disk, and blood vessels). Four deep neural networks, including VGG-11, ResNet-18, DeiT-Tiny, and Swin Transformer-Tiny, were developed using binary diagnostic labels of glaucoma and five CAM methods (Grad-CAM, XGrad-CAM, Score-CAM, Eigen-CAM, and Layer-CAM) were employed to highlight the model focus area. We applied the paired-sample t-test to compare the percentage of anatomies in the model focus area to the proportion of anatomies in the entire image. After that, Pearson's and Spearman's correlation tests were implemented to examine the relationship between model predictive ability and the percentage of anatomical structures in the model focus area. On five public glaucoma datasets, all deep learning models consistently displayed statistically significantly higher percentages of anatomical structures in the focus area than the proportions of anatomical structures in the entire image. Also, we validated the positive relationship between the percentage of anatomical structures in the focus area and model predictive performance. Our study provides evidence of the convergence of decision logic between deep neural networks and human clinicians through rigorous statistical tests. We anticipate that it can help alleviate clinicians' concerns regarding the trustworthiness of deep learning in healthcare. For reproducibility, the code and dataset have been released at GitHub.