Fine-Tuning a Local LLaMA-3 Large Language Model for Automated Privacy-Preserving Physician Letter Generation in Radiation Oncology

TL;DR

Problem: automating physician letter generation under privacy constraints in radiation oncology. Approach: locally fine-tune LLaMA-3-8B with QLoRA on hospital data using institution-specific letters and case summaries; outputs in German for letters and English for summaries. Key findings: base LLaMA models are inadequate; the 8B LLaMA-3 model achieved institution-specific content and better ROUGE scores than the 13B LLaMA-2 baseline; expert evaluations indicate practical utility with need for physician review. Significance: demonstrates privacy-preserving, locally deployable AI-assisted documentation in radiation oncology and feasible resource usage on a single 48 GB GPU.

Abstract

Generating physician letters is a time-consuming task in daily clinical practice. This study investigates local fine-tuning of large language models (LLMs), specifically LLaMA models, for physician letter generation in a privacy-preserving manner within the field of radiation oncology. Our findings demonstrate that base LLaMA models, without fine-tuning, are inadequate for effectively generating physician letters. The QLoRA algorithm provides an efficient method for local intra-institutional fine-tuning of LLMs with limited computational resources (i.e., a single 48 GB GPU workstation within the hospital). The fine-tuned LLM successfully learns radiation oncology-specific information and generates physician letters in an institution-specific style. ROUGE scores of the generated summary reports highlight the superiority of the 8B LLaMA-3 model over the 13B LLaMA-2 model. Further multidimensional physician evaluations of 10 cases reveal that, although the fine-tuned LLaMA-3 model has limited capacity to generate content beyond the provided input data, it successfully generates salutations, diagnoses and treatment histories, recommendations for further treatment, and planned schedules. Overall, clinical benefit was rated highly by the clinical experts (average score of 3.44 on a 4-point scale). With careful physician review and correction, automated LLM-based physician letter generation has significant practical value.

Figures (5)

• Figure 1: The outputs of the locally fine-tuned LLaMA-2 (center) and LLaMA-3 (bottom) models compared to the baseline LLaMA-2 model in an exemplary case for the task of patient case summarisation.
• Figure 2: The ROUGE scores of LLaMA models for the task of patient case summarisation with and without local fine-tuning on institutional data. The error bars indicate standard deviations.
• Figure 3: The distribution of average physician rating scores for the physician letters automatically generated by the locally fine-tuned 8B LLaMA-3 model. The error bars indicate standard deviations.
• Figure 4: Input medical data of Case $\#$1 for the physician letter generation task. Note that some keywords are highlighted in bold by the authors for better visualization, but the content was provided in plain text to the LLM. Certain private information is anonymized with the symbol $\ast$. Different segments of the patient input information in regard to the model output (Fig. \ref{['Fig:OutputExample2']}) are highlighted by different colors.
• Figure 5: Fine-tuned 8B LLaMA-3 model output of Case $\#$1 for the physician letter generation and its corresponding reference output. The patient name is anonymized with the symbol $\ast$. The highlighted text segments correspond to the information in the input data (Fig. \ref{['Fig:InputExample2']}) highlighted with the same color.