Uncertainty Quantification in Detecting Choroidal Metastases on MRI via Evolutionary Strategies

TL;DR

It is found that subjective features appreciated by human radiologists explained images for which uncertainty was high, highlighting the significance of uncertainty quantification in AI-driven radiological analyses.

Abstract

Uncertainty quantification plays a vital role in facilitating the practical implementation of AI in radiology by addressing growing concerns around trustworthiness. Given the challenges associated with acquiring large, annotated datasets in this field, there is a need for methods that enable uncertainty quantification in small data AI approaches tailored to radiology images. In this study, we focused on uncertainty quantification within the context of the small data evolutionary strategies-based technique of deep neuroevolution (DNE). Specifically, we employed DNE to train a simple Convolutional Neural Network (CNN) with MRI images of the eyes for binary classification. The goal was to distinguish between normal eyes and those with metastatic tumors called choroidal metastases. The training set comprised 18 images with choroidal metastases and 18 without tumors, while the testing set contained a tumor-to-normal ratio of 15:15. We trained CNN model weights via DNE for approximately 40,000 episodes, ultimately reaching a convergence of 100% accuracy on the training set. We saved all models that achieved maximal training set accuracy. Then, by applying these models to the testing set, we established an ensemble method for uncertainty quantification.The saved set of models produced distributions for each testing set image between the two classes of normal and tumor-containing. The relative frequencies permitted uncertainty quantification of model predictions. Intriguingly, we found that subjective features appreciated by human radiologists explained images for which uncertainty was high, highlighting the significance of uncertainty quantification in AI-driven radiological analyses.

Figures (6)

• Figure 1: Multi-input convolutional neural network architecture used for the binary classification (tumor vs normal) of choroidal metastases from Brain MRI slices. The network takes as input two grayscale images, each of size (40, 40, 1) : (Height, Width, Channel), and produces a Softmax distribution over the respective classes of interest. Note that the bidirectional arrows represent shared weights between blocks. Respective convolutional channels (for CNN layers) and Neuron counts (Linear layers) are marked against each block. Respective hyperparameters used in training the network as per Algorithm \ref{['alg:init-uq-algorithm']} : {$\alpha = 0.12,\; p = 40,\; N_{epochs} = 10^5, \;R_{max}^{\mathcal{D}_T } = 30\; or \; 36$ }
• Figure 2: Accuracy versus training time in generations during training of the CNN from Figure \ref{['arch_2']}. This is an average formed from 5 repeated runs of one of the 2-fold cross-validation training runs, with a training set size of 15 and testing set size of 18.
• Figure 3: Examples of different uncertainties in tumor-containing cases. The top row displays an example from class 1, in which there is a correct prediction with low uncertainty (defined as being less than 20 %). Class 2, in the middle row, depicts a high-uncertainty scenario. We can see that the tumor is volume-averaged with the adjacent high T2 signal aqueous humor. As such the lesion is less contrastive within the image. Class 3, with an incorrect prediction of normal with low uncertainty, shows a small plaque-like lesion in the left globe that could easily be overlooked by a radiologist.
• Figure 4: Examples of different uncertainties in normal, tumor-free cases. The top row displays an example from class 1, in which there is a correct prediction with low uncertainty (defined as being less than 20 %). Class 2, in the middle row, depicts a high-uncertainty scenario. We can see that that the motion and pulsation artifact creates a vague structure that could fool the algorithm into believing it may be a lesion. The class 3 example, which incorrectly predicts a tumor with low unlikelihood, displays apparent thickening at the posterior globe. Although this is due to volume averaging and slice selection, it can be seen to resemble a mass to the AI algorithm.
• Figure 5: Learning curve during training of the CNN with gradient-based optimization and dropout (Monte Carlo approach).