Crowdsourcing with Enhanced Data Quality Assurance: An Efficient Approach to Mitigate Resource Scarcity Challenges in Training Large Language Models for Healthcare

TL;DR

This work tackles data scarcity in healthcare LLMs by introducing a crowdsourcing framework with pre-, real-time-, and post-data quality control. Real-time QC, leveraging copy-detection and search-based verification, yields substantial data-quality gains and improved precision, while affecting recall depending on the QC stage. Fine-tuning Bio-BERT with crowdsourced data increases recall but often reduces precision, with real-time QC offering the most favorable balance, and post-QC adding limited benefits at a higher cost. The study demonstrates the value of structured QC in resource-constrained healthcare NLP, offering practical guidance for constructing higher-quality labeled data for clinical decision support systems.

Abstract

Large Language Models (LLMs) have demonstrated immense potential in artificial intelligence across various domains, including healthcare. However, their efficacy is hindered by the need for high-quality labeled data, which is often expensive and time-consuming to create, particularly in low-resource domains like healthcare. To address these challenges, we propose a crowdsourcing (CS) framework enriched with quality control measures at the pre-, real-time-, and post-data gathering stages. Our study evaluated the effectiveness of enhancing data quality through its impact on LLMs (Bio-BERT) for predicting autism-related symptoms. The results show that real-time quality control improves data quality by 19 percent compared to pre-quality control. Fine-tuning Bio-BERT using crowdsourced data generally increased recall compared to the Bio-BERT baseline but lowered precision. Our findings highlighted the potential of crowdsourcing and quality control in resource-constrained environments and offered insights into optimizing healthcare LLMs for informed decision-making and improved patient care.