Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BURExtract-Llama: An LLM for Clinical Concept Extraction in Breast Ultrasound Reports

Yuxuan Chen, Haoyan Yang, Hengkai Pan, Fardeen Siddiqui, Antonio Verdone, Qingyang Zhang, Sumit Chopra, Chen Zhao, Yiqiu Shen

TL;DR

This work tackles extracting structured clinical concepts from breast ultrasound reports by proposing an in-house LLM pipeline. It labels a small dataset with GPT-4-32K and fine-tunes a Llama-3-8B model using QLoRA to output JSON representations of lesions across 16 attributes. On 4000 NYU Langone reports, BURExtract-Llama achieves an average F1 around 84.6% and can match GPT-4's performance while delivering 2-second inferences and preserving data privacy. The approach demonstrates a practical, cost-effective path for privacy-preserving clinical NLP in radiology, though external validation and label-noise considerations remain for future work.

Abstract

Breast ultrasound is essential for detecting and diagnosing abnormalities, with radiology reports summarizing key findings like lesion characteristics and malignancy assessments. Extracting this critical information is challenging due to the unstructured nature of these reports, with varied linguistic styles and inconsistent formatting. While proprietary LLMs like GPT-4 are effective, they are costly and raise privacy concerns when handling protected health information. This study presents a pipeline for developing an in-house LLM to extract clinical information from radiology reports. We first use GPT-4 to create a small labeled dataset, then fine-tune a Llama3-8B model on it. Evaluated on clinician-annotated reports, our model achieves an average F1 score of 84.6%, which is on par with GPT-4. Our findings demonstrate the feasibility of developing an in-house LLM that not only matches GPT-4's performance but also offers cost reductions and enhanced data privacy.

BURExtract-Llama: An LLM for Clinical Concept Extraction in Breast Ultrasound Reports

TL;DR

This work tackles extracting structured clinical concepts from breast ultrasound reports by proposing an in-house LLM pipeline. It labels a small dataset with GPT-4-32K and fine-tunes a Llama-3-8B model using QLoRA to output JSON representations of lesions across 16 attributes. On 4000 NYU Langone reports, BURExtract-Llama achieves an average F1 around 84.6% and can match GPT-4's performance while delivering 2-second inferences and preserving data privacy. The approach demonstrates a practical, cost-effective path for privacy-preserving clinical NLP in radiology, though external validation and label-noise considerations remain for future work.

Abstract

Breast ultrasound is essential for detecting and diagnosing abnormalities, with radiology reports summarizing key findings like lesion characteristics and malignancy assessments. Extracting this critical information is challenging due to the unstructured nature of these reports, with varied linguistic styles and inconsistent formatting. While proprietary LLMs like GPT-4 are effective, they are costly and raise privacy concerns when handling protected health information. This study presents a pipeline for developing an in-house LLM to extract clinical information from radiology reports. We first use GPT-4 to create a small labeled dataset, then fine-tune a Llama3-8B model on it. Evaluated on clinician-annotated reports, our model achieves an average F1 score of 84.6%, which is on par with GPT-4. Our findings demonstrate the feasibility of developing an in-house LLM that not only matches GPT-4's performance but also offers cost reductions and enhanced data privacy.
Paper Structure (23 sections, 17 equations, 3 figures, 3 tables)

This paper contains 23 sections, 17 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methods
  4. Problem Formulation
  5. Pipeline for building in-house LLMs
  6. Extract Observations and Impressions
  7. Generate Training Labels using GPT-4-32K
  8. Fine-Tuning
  9. Inference
  10. Experiments
  11. Dataset
  12. Training Detail
  13. Evaluation Metrics
  14. Results
  15. Error Analysis
  16. ...and 8 more sections

Figures (3)

  • Figure 1: The pipeline for building and utilizing the BURExtract-Llama. Steps 1-3 cover the building process: extracting observations and impressions, generating training labels with GPT-4, and fine-tuning BURExtract-Llama. Step 4 shows how BURExtract-Llama infers from new reports and outputs structured JSON.
  • Figure :
  • Figure :