Analysis of Incursive Breast Cancer in Mammograms Using YOLO, Explainability, and Domain Adaptation
Jayan Adhikari, Prativa Joshi, Susish Baral
TL;DR
The paper tackles reliability challenges of AI-based breast cancer detection when presented with Out-of-Distribution inputs. It proposes a two-stage pipeline that first filters inputs using a ResNet50-based OOD detector with a cosine similarity gallery, then applies YOLOv8 for lesion detection with Grad-CAM explanations. The method selects ResNet50 after an architectural search and demonstrates high OOD accuracy (99.77%) and strong detection performance (mAP@0.5 = 0.947) on INbreast data with an OOD Kaggle set for validation. The work emphasizes interpretability and domain robustness, offering an open-source PyPI implementation and a foundation for deployment across diverse clinical environments.
Abstract
Deep learning models for breast cancer detection from mammographic images have significant reliability problems when presented with Out-of-Distribution (OOD) inputs such as other imaging modalities (CT, MRI, X-ray) or equipment variations, leading to unreliable detection and misdiagnosis. The current research mitigates the fundamental OOD issue through a comprehensive approach integrating ResNet50-based OOD filtering with YOLO architectures (YOLOv8, YOLOv11, YOLOv12) for accurate detection of breast cancer. Our strategy establishes an in-domain gallery via cosine similarity to rigidly reject non-mammographic inputs prior to processing, ensuring that only domain-associated images supply the detection pipeline. The OOD detection component achieves 99.77\% general accuracy with immaculate 100\% accuracy on OOD test sets, effectively eliminating irrelevant imaging modalities. ResNet50 was selected as the optimum backbone after 12 CNN architecture searches. The joint framework unites OOD robustness with high detection performance (mAP@0.5: 0.947) and enhanced interpretability through Grad-CAM visualizations. Experimental validation establishes that OOD filtering significantly improves system reliability by preventing false alarms on out-of-distribution inputs while maintaining higher detection accuracy on mammographic data. The present study offers a fundamental foundation for the deployment of reliable AI-based breast cancer detection systems in diverse clinical environments with inherent data heterogeneity.
