Unmasking unlearnable models: a classification challenge for biomedical images without visible cues

TL;DR

This paper tackles the challenge of predicting MGMT methylation status from MRI images that lack visually discernible cues, a problem with high clinical relevance due to treatment decisions in glioma. Through a process-driven evaluation, the authors benchmark multiple CNN architectures, explore transfer learning, analyze gradient flow and learning curves, and apply radiomics-based feature selection to probe the roots of limited learnability. They find that current models are largely unlearnable for this task, with transfer learning offering limited gains and complexity unable to overcome fundamental limitations. Radiomic features provide some informative signal, but are not sufficient alone, prompting the authors to advocate domain-informed feature maps and indigenous architectural strategies to address non-visible cues and improve real-world applicability.

Abstract

Predicting traits from images lacking visual cues is challenging, as algorithms are designed to capture visually correlated ground truth. This problem is critical in biomedical sciences, and their solution can improve the efficacy of non-invasive methods. For example, a recent challenge of predicting MGMT methylation status from MRI images is critical for treatment decisions of glioma patients. Using less robust models poses a significant risk in these critical scenarios and underscores the urgency of addressing this issue. Despite numerous efforts, contemporary models exhibit suboptimal performance, and underlying reasons for this limitation remain elusive. In this study, we demystify the complexity of MGMT status prediction through a comprehensive exploration by performing benchmarks of existing models adjoining transfer learning. Their architectures were further dissected by observing gradient flow across layers. Additionally, a feature selection strategy was applied to improve model interpretability. Our finding highlighted that current models are unlearnable and may require new architectures to explore applications in the real world. We believe our study will draw immediate attention and catalyse advancements in predictive modelling with non-visible cues.

Figures (5)

• Figure 1: Comparison of hyper parameters (A) Testing different combinations of loss function and error metric on training. (B) Testing different combinations of loss function and error metric on test dataset. In both the cases the accuracy for Hinge with RMS and Adam has been overlapped.
• Figure 2: The various transfer learning approach using MedNet weights (A-B) The MedNet weight used as feature extractor and fully connected layer are trained using current data in training and test split. (C-D) The MedNet weight was fine-tuned using the current dataset and performance evaluated on training and test data.
• Figure 3: The model investigation using learning curve with sensitivity and specificity. (A-B) The fluctuation of sensitivity and specificity in training and test dataset.
• Figure 4: The model investigation using gradient flow. (A) The model weight in the primary layer of ResNet. The figure shows the histogram for different epochs with learned weights. (B) The gradient updates for subsequent epochs of training. The figure shows the histogram for different epochs with calculated gradient of weights.
• Figure 5: Determining important radiomic feature between MGMT methylated and non-methylated category. (A) Statistically significant features for different image modalities. (B) Common feature among the image modality (T1, T1CE, FLAIR), Here image modality T2 has not been considered as it has only one significant feature, which was not common in any other modality. (C) Feature ranking using recursive feature elimination with random forest. (D) Clustering of similar feature using dendrogram, showing three clusters with two features each.