High-Fidelity Pseudo-label Generation by Large Language Models for Training Robust Radiology Report Classifiers

TL;DR

The paper addresses the bottleneck of labeling chest X-ray reports by combining powerful LLM-based pseudo-labeling with knowledge distillation into a DeBERTa-Base student, enabling high-accuracy, high-throughput radiology report classification across 13 findings with Present/Absent/Uncertain statuses. The two-stage DeBERTa-RAD framework achieves a state-of-the-art Macro F1 of $0.9120$ on the MIMIC-500 benchmark while maintaining efficient inference (~$750$ reports/sec), significantly reducing reliance on manual annotations and prohibitive LLM inference costs. Key contributions include high-quality pseudo-label generation, a robust DeBERTa-based student trained via a tailored distillation objective, and extensive evaluation including human expert validation, showing improved handling of uncertainty and negation. The approach offers a practical path for scalable clinical NLP tasks and can extend to other radiology or medical text labeling challenges through domain-specific pseudo-labels and distillation strategies.

Abstract

Automated labeling of chest X-ray reports is essential for enabling downstream tasks such as training image-based diagnostic models, population health studies, and clinical decision support. However, the high variability, complexity, and prevalence of negation and uncertainty in these free-text reports pose significant challenges for traditional Natural Language Processing methods. While large language models (LLMs) demonstrate strong text understanding, their direct application for large-scale, efficient labeling is limited by computational cost and speed. This paper introduces DeBERTa-RAD, a novel two-stage framework that combines the power of state-of-the-art LLM pseudo-labeling with efficient DeBERTa-based knowledge distillation for accurate and fast chest X-ray report labeling. We leverage an advanced LLM to generate high-quality pseudo-labels, including certainty statuses, for a large corpus of reports. Subsequently, a DeBERTa-Base model is trained on this pseudo-labeled data using a tailored knowledge distillation strategy. Evaluated on the expert-annotated MIMIC-500 benchmark, DeBERTa-RAD achieves a state-of-the-art Macro F1 score of 0.9120, significantly outperforming established rule-based systems, fine-tuned transformer models, and direct LLM inference, while maintaining a practical inference speed suitable for high-throughput applications. Our analysis shows particular strength in handling uncertain findings. This work demonstrates a promising path to overcome data annotation bottlenecks and achieve high-performance medical text processing through the strategic combination of LLM capabilities and efficient student models trained via distillation.