Effectiveness of Automatically Curated Dataset in Thyroid Nodules Classification Algorithms Using Deep Learning

TL;DR

This study addresses data scarcity in deep learning for thyroid nodule ultrasound classification by evaluating an automatic labeling pipeline, MADLaP, against manually curated data. Using identical CNN architecture across three datasets (Manual Set, MADLaP Set, and S1 Set) and an independent test set, the authors show that models trained on MADLaP data achieve higher AUC than those trained on manually labeled data ($\text{AUC}$ up to $0.694$ versus $0.643$, respectively). The comparison between MADLaP and its Stage 1 subset indicates similar performance, while batch-size experiments reveal that larger batches mitigate the impact of label noise, reinforcing the value of using the full MADLaP output. Overall, the findings support deploying automatically-curated data with appropriate training settings to substantially boost deep learning performance in thyroid nodule classification, reducing the reliance on extensive manual annotation.

Abstract

The diagnosis of thyroid nodule cancers commonly utilizes ultrasound images. Several studies showed that deep learning algorithms designed to classify benign and malignant thyroid nodules could match radiologists' performance. However, data availability for training deep learning models is often limited due to the significant effort required to curate such datasets. The previous study proposed a method to curate thyroid nodule datasets automatically. It was tested to have a 63% yield rate and 83% accuracy. However, the usefulness of the generated data for training deep learning models remains unknown. In this study, we conducted experiments to determine whether using a automatically-curated dataset improves deep learning algorithms' performance. We trained deep learning models on the manually annotated and automatically-curated datasets. We also trained with a smaller subset of the automatically-curated dataset that has higher accuracy to explore the optimum usage of such dataset. As a result, the deep learning model trained on the manually selected dataset has an AUC of 0.643 (95% confidence interval [CI]: 0.62, 0.66). It is significantly lower than the AUC of the 6automatically-curated dataset trained deep learning model, 0.694 (95% confidence interval [CI]: 0.67, 0.73, P < .001). The AUC of the accurate subset trained deep learning model is 0.689 (95% confidence interval [CI]: 0.66, 0.72, P > .43), which is insignificantly worse than the AUC of the full automatically-curated dataset. In conclusion, we showed that using a automatically-curated dataset can substantially increase the performance of deep learning algorithms, and it is suggested to use all the data rather than only using the accurate subset.

Figures (3)

• Figure 1: Experimental design and algorithm comparison
• Figure 2: Structure of the deep convolutional neural network used for malignancy classification. Dimensions denote feature map sizes after each operation.
• Figure 3: Performance in respect to batch sizes for three training datasets