Informed Deep Abstaining Classifier: Investigating noise-robust training for diagnostic decision support systems

TL;DR

The noise-robust Deep Abstaining Classifier loss is expanded to an Informed Deep Abstaining Classifier (IDAC) loss by incorporating noise level estimations during training to enhance the noise robustness compared to DAC and several state-of-the-art loss functions.

Abstract

Image-based diagnostic decision support systems (DDSS) utilizing deep learning have the potential to optimize clinical workflows. However, developing DDSS requires extensive datasets with expert annotations and is therefore costly. Leveraging report contents from radiological data bases with Natural Language Processing to annotate the corresponding image data promises to replace labor-intensive manual annotation. As mining "real world" databases can introduce label noise, noise-robust training losses are of great interest. However, current noise-robust losses do not consider noise estimations that can for example be derived based on the performance of the automatic label generator used. In this study, we expand the noise-robust Deep Abstaining Classifier (DAC) loss to an Informed Deep Abstaining Classifier (IDAC) loss by incorporating noise level estimations during training. Our findings demonstrate that IDAC enhances the noise robustness compared to DAC and several state-of-the-art loss functions. The results are obtained on various simulated noise levels using a public chest X-ray data set. These findings are reproduced on an in-house noisy data set, where labels were extracted from the clinical systems of the University Hospital Bonn by a text-based transformer. The IDAC can therefore be a valuable tool for researchers, companies or clinics aiming to develop accurate and reliable DDSS from routine clinical data.

Figures (3)

• Figure 1: AUROC scores (%) on the CheXpert test set for simulated noise levels below 50% for the detection of pleural effusion. The proposed IDAC loss achieves the highest noise robustness. Note that we only visualize scores higher than 80% due to overview reasons.
• Figure 2: AUROC scores (%) on the CheXpert test set for simulated noise levels below 50% for the detection of cardiomegaly. The proposed IDAC loss achieves the highest noise robustness.
• Figure 3: Left: Smoothed validation performance of proposed IDAC training for classifying pleural effusion for CheXpert. Different abstention weights $\alpha$ for estimated noise levels $\Tilde{\eta}=1\%$ (upper) and $\Tilde{\eta}=30\%$ (lower) are investigated. Warm-up without abstention is 740 steps. For comparison, training with CE is illustrated in yellow and with DAC in purple. Right: Smoothed abstention rate of IDAC and DAC training with the different weight parameters $\alpha$ for the noise levels $1\%$ (upper) and $30\%$ (lower). The loss abstains from a training case when the model outputs the highest probability on the $p_{k+1}$ abstention neuron after softmax.