Fine-Tuning In-House Large Language Models to Infer Differential Diagnosis from Radiology Reports
Luoyao Chen, Revant Teotia, Antonio Verdone, Aidan Cardall, Lakshay Tyagi, Yiqiu Shen, Sumit Chopra
TL;DR
The paper tackles automatic extraction of differential diagnoses from radiology report impressions and proposes an in-house LLM fine-tuning pipeline to address privacy and cost concerns of proprietary models. It uses GPT-4-32K to generate labels for 31,056 musculoskeletal radiology impressions and then fine-tunes open-source LLMs with QLoRa, achieving 92.1% average F1 and competitive recall relative to GPT-4 while using only about 5% trainable parameters. The approach is validated on 1,067 clinician-annotated tests across 133 pathologies and shows robust performance across CT, MRI, and CR modalities, suggesting practical, privacy-preserving deployment. The method offers a scalable alternative to proprietary LLMs and can generalize to other radiology sections, enabling cost savings and enhanced PHI security.
Abstract
Radiology reports summarize key findings and differential diagnoses derived from medical imaging examinations. The extraction of differential diagnoses is crucial for downstream tasks, including patient management and treatment planning. However, the unstructured nature of these reports, characterized by diverse linguistic styles and inconsistent formatting, presents significant challenges. Although proprietary large language models (LLMs) such as GPT-4 can effectively retrieve clinical information, their use is limited in practice by high costs and concerns over the privacy of protected health information (PHI). This study introduces a pipeline for developing in-house LLMs tailored to identify differential diagnoses from radiology reports. We first utilize GPT-4 to create 31,056 labeled reports, then fine-tune open source LLM using this dataset. Evaluated on a set of 1,067 reports annotated by clinicians, the proposed model achieves an average F1 score of 92.1\%, which is on par with GPT-4 (90.8\%). Through this study, we provide a methodology for constructing in-house LLMs that: match the performance of GPT, reduce dependence on expensive proprietary models, and enhance the privacy and security of PHI.