Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MRScore: Evaluating Radiology Report Generation with LLM-based Reward System

Yunyi Liu, Zhanyu Wang, Yingshu Li, Xinyu Liang, Lingqiao Liu, Lei Wang, Luping Zhou

TL;DR

The paper tackles the mismatch between traditional NLG metrics and radiologist judgments in automated radiology report generation. It introduces MRScore, an LLM-based reward model trained with RLHF, built on seven criteria, with GPT-4-generated evaluation samples guiding training; scoring leverages a paired accepted/rejected scheme and a LoRA-finetuned backbone, yielding a final reward score. MRScore demonstrates higher correlation with human judgments than standard metrics such as BLEU, ROUGE, METEOR, CIDEr, BERTScore, RadGraph F1, and RadCliQ across a GPT-4V evaluation dataset drawn from MIMIC-CXR. This approach enables scalable, cost-effective, and transparent evaluation of radiology reports, with code and data to be released on GitHub.

Abstract

In recent years, automated radiology report generation has experienced significant growth. This paper introduces MRScore, an automatic evaluation metric tailored for radiology report generation by leveraging Large Language Models (LLMs). Conventional NLG (natural language generation) metrics like BLEU are inadequate for accurately assessing the generated radiology reports, as systematically demonstrated by our observations within this paper. To address this challenge, we collaborated with radiologists to develop a framework that guides LLMs for radiology report evaluation, ensuring alignment with human analysis. Our framework includes two key components: i) utilizing GPT to generate large amounts of training data, i.e., reports with different qualities, and ii) pairing GPT-generated reports as accepted and rejected samples and training LLMs to produce MRScore as the model reward. Our experiments demonstrate MRScore's higher correlation with human judgments and superior performance in model selection compared to traditional metrics. Our code and datasets will be available on GitHub.

MRScore: Evaluating Radiology Report Generation with LLM-based Reward System

TL;DR

The paper tackles the mismatch between traditional NLG metrics and radiologist judgments in automated radiology report generation. It introduces MRScore, an LLM-based reward model trained with RLHF, built on seven criteria, with GPT-4-generated evaluation samples guiding training; scoring leverages a paired accepted/rejected scheme and a LoRA-finetuned backbone, yielding a final reward score. MRScore demonstrates higher correlation with human judgments than standard metrics such as BLEU, ROUGE, METEOR, CIDEr, BERTScore, RadGraph F1, and RadCliQ across a GPT-4V evaluation dataset drawn from MIMIC-CXR. This approach enables scalable, cost-effective, and transparent evaluation of radiology reports, with code and data to be released on GitHub.

Abstract

In recent years, automated radiology report generation has experienced significant growth. This paper introduces MRScore, an automatic evaluation metric tailored for radiology report generation by leveraging Large Language Models (LLMs). Conventional NLG (natural language generation) metrics like BLEU are inadequate for accurately assessing the generated radiology reports, as systematically demonstrated by our observations within this paper. To address this challenge, we collaborated with radiologists to develop a framework that guides LLMs for radiology report evaluation, ensuring alignment with human analysis. Our framework includes two key components: i) utilizing GPT to generate large amounts of training data, i.e., reports with different qualities, and ii) pairing GPT-generated reports as accepted and rejected samples and training LLMs to produce MRScore as the model reward. Our experiments demonstrate MRScore's higher correlation with human judgments and superior performance in model selection compared to traditional metrics. Our code and datasets will be available on GitHub.
Paper Structure (9 sections, 2 equations, 2 figures, 4 tables)

This paper contains 9 sections, 2 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Scoring Criteria Design and Scoring Dataset Generation
  4. LLM-based Reward Model
  5. Loss Function
  6. Experiments and Result
  7. Evaluation Dataset
  8. Experiment Result
  9. Conclusion

Figures (2)

  • Figure 1: Overview of MRScore. The upper portion illustrates the data generation process, while the lower portion represents the training process for the reward model using LoRA. In the lower portion, the solid line indicates the training phase while the dashed line indicates the inference phase.
  • Figure 2: Correlations between varied evaluation metrics and radiologist scores