Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation

Zilong Wang, Xufang Luo, Xinyang Jiang, Dongsheng Li, Lili Qiu

TL;DR

This work introduces LLM-RadJudge, an evaluation framework that leverages large language models to assess radiology reports with radiologist-level alignment. It demonstrates that GPT-4, with a two-stage prompting strategy, achieves Kendall's tau around $0.735$, closely matching radiologist judgments and outperforming conventional metrics. To address cost and speed, the authors construct a large-scale dataset from LLM evaluations and distill the capability into smaller models (notably BioMistral-7B), achieving $ au \,\approx\,0.749$ while offering faster inference. The approach enables practical, scalable evaluation of radiology report generation and is intended to be open-sourced, facilitating broader adoption and ongoing improvements in clinically relevant model development.

Abstract

Evaluating generated radiology reports is crucial for the development of radiology AI, but existing metrics fail to reflect the task's clinical requirements. This study proposes a novel evaluation framework using large language models (LLMs) to compare radiology reports for assessment. We compare the performance of various LLMs and demonstrate that, when using GPT-4, our proposed metric achieves evaluation consistency close to that of radiologists. Furthermore, to reduce costs and improve accessibility, making this method practical, we construct a dataset using LLM evaluation results and perform knowledge distillation to train a smaller model. The distilled model achieves evaluation capabilities comparable to GPT-4. Our framework and distilled model offer an accessible and efficient evaluation method for radiology report generation, facilitating the development of more clinically relevant models. The model will be further open-sourced and accessible.

LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation

TL;DR

This work introduces LLM-RadJudge, an evaluation framework that leverages large language models to assess radiology reports with radiologist-level alignment. It demonstrates that GPT-4, with a two-stage prompting strategy, achieves Kendall's tau around , closely matching radiologist judgments and outperforming conventional metrics. To address cost and speed, the authors construct a large-scale dataset from LLM evaluations and distill the capability into smaller models (notably BioMistral-7B), achieving while offering faster inference. The approach enables practical, scalable evaluation of radiology report generation and is intended to be open-sourced, facilitating broader adoption and ongoing improvements in clinically relevant model development.

Abstract

Evaluating generated radiology reports is crucial for the development of radiology AI, but existing metrics fail to reflect the task's clinical requirements. This study proposes a novel evaluation framework using large language models (LLMs) to compare radiology reports for assessment. We compare the performance of various LLMs and demonstrate that, when using GPT-4, our proposed metric achieves evaluation consistency close to that of radiologists. Furthermore, to reduce costs and improve accessibility, making this method practical, we construct a dataset using LLM evaluation results and perform knowledge distillation to train a smaller model. The distilled model achieves evaluation capabilities comparable to GPT-4. Our framework and distilled model offer an accessible and efficient evaluation method for radiology report generation, facilitating the development of more clinically relevant models. The model will be further open-sourced and accessible.
Paper Structure (16 sections, 5 figures, 1 table)

This paper contains 16 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation Metrics for Radiology Reports
  4. Language metrics
  5. Clinical metrics
  6. Large Language Model for Evaluation
  7. Report Evaluation Using Large Language Models (LLMs)
  8. Motivating Examples
  9. Architecting LLM-Based Methodology for Radiology Report Assessment
  10. Benchmarking LLMs Alignments with Radiologist Expertise
  11. Achieving Radiologist-Level Evaluation with LLM
  12. Developing an Efficient Model for Accessible Evaluation
  13. Curating Dataset for Model Training
  14. Knowledge Distillation: Methodology and Experimental Setup
  15. Evaluating the Performance of the Efficient Model
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Two group of cases of reference reports and generated reports
  • Figure 2: Accordance between LLMs and radiologists (A) is GPT-4 with single step prompt; (B)-(J) are different LLMs generated results; (K) is fine-tuned Mistral-7B; (L) is fine-tuned BioMistral-7B
  • Figure 3: Error distribution of LLMs Error = LLM evaluation - average counts of six radiologist
  • Figure 4: Accordance between radiologists
  • Figure 5: Comparison between GPT4 and finetuned models