QUAD-LLM-MLTC: Large Language Models Ensemble Learning for Healthcare Text Multi-Label Classification
Hajar Sakai, Sarah S. Lam
TL;DR
This work tackles healthcare text multi-label classification under limited labeled data by introducing QUAD-LLM-MLTC, an unsupervised ensemble that integrates four LLMs (GPT-4o, BERT, PEGASUS, BART) in a sequential pipeline with token extraction, augmentation, classification, and probability estimation, then merging outputs via stacking. It formalizes MLTC, details prompt-driven base classifiers, and implements a Lin-SVM with Classifier Chains as the meta-learner, validated on the Hallmarks of Cancer HoC dataset with stratified samples. Results show QUAD-LLM-MLTC consistently outperforms traditional baselines and single-model prompts in F1 and AUC across topics and dataset sizes, with strong statistical support and robust cross-validation. The approach offers a scalable, annotation-light solution for rapid healthcare text categorization, while noting propagation of first-stage errors and the potential gains from targeted fine-tuning or broader datasets as future work.
Abstract
The escalating volume of collected healthcare textual data presents a unique challenge for automated Multi-Label Text Classification (MLTC), which is primarily due to the scarcity of annotated texts for training and their nuanced nature. Traditional machine learning models often fail to fully capture the array of expressed topics. However, Large Language Models (LLMs) have demonstrated remarkable effectiveness across numerous Natural Language Processing (NLP) tasks in various domains, which show impressive computational efficiency and suitability for unsupervised learning through prompt engineering. Consequently, these LLMs promise an effective MLTC of medical narratives. However, when dealing with various labels, different prompts can be relevant depending on the topic. To address these challenges, the proposed approach, QUAD-LLM-MLTC, leverages the strengths of four LLMs: GPT-4o, BERT, PEGASUS, and BART. QUAD-LLM-MLTC operates in a sequential pipeline in which BERT extracts key tokens, PEGASUS augments textual data, GPT-4o classifies, and BART provides topics' assignment probabilities, which results in four classifications, all in a 0-shot setting. The outputs are then combined using ensemble learning and processed through a meta-classifier to produce the final MLTC result. The approach is evaluated using three samples of annotated texts, which contrast it with traditional and single-model methods. The results show significant improvements across the majority of the topics in the classification's F1 score and consistency (F1 and Micro-F1 scores of 78.17% and 80.16% with standard deviations of 0.025 and 0.011, respectively). This research advances MLTC using LLMs and provides an efficient and scalable solution to rapidly categorize healthcare-related text data without further training.