Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReFINE: A Reward-Based Framework for Interpretable and Nuanced Evaluation of Radiology Report Generation

Yunyi Liu, Yingshu Li, Zhanyu Wang, Xinyu Liang, Lingqiao Liu, Lei Wang, Luping Zhou

TL;DR

ReFINE tackles the misalignment between automated radiology report evaluation and human judgment by introducing a reward-based metric trained with an LLM reward model and a margin-based loss. It leverages GPT-4 to generate labeled training samples across two scoring frameworks (RadCliQ and MRScore) and fine-tunes a LoRA-enabled Llama3 to output per-criterion rewards that sum to an overall score. The approach yields higher alignment with radiologists than traditional metrics and offers interpretable sub-scores that pinpoint specific errors. Its offline deployment and criterion flexibility make ReFINE a scalable, privacy-conscious tool for evaluating radiology reports with practical clinical impact.

Abstract

Automated radiology report generation (R2Gen) has advanced significantly, introducing challenges in accurate evaluation due to its complexity. Traditional metrics often fall short by relying on rigid word-matching or focusing only on pathological entities, leading to inconsistencies with human assessments. To bridge this gap, we introduce ReFINE, an automatic evaluation metric designed specifically for R2Gen. Our metric utilizes a reward model, guided by our margin-based reward enforcement loss, along with a tailored training data design that enables customization of evaluation criteria to suit user-defined needs. It not only scores reports according to user-specified criteria but also provides detailed sub-scores, enhancing interpretability and allowing users to adjust the criteria between different aspects of reports. Leveraging GPT-4, we designed an easy-to-use data generation pipeline, enabling us to produce extensive training data based on two distinct scoring systems, each containing reports of varying quality along with corresponding scores. These GPT-generated reports are then paired as accepted and rejected samples through our pairing rule to train an LLM towards our fine-grained reward model, which assigns higher rewards to the report with high quality. Our reward-control loss enables this model to simultaneously output multiple individual rewards corresponding to the number of evaluation criteria, with their summation as our final ReFINE. Our experiments demonstrate ReFINE's heightened correlation with human judgments and superior performance in model selection compared to traditional metrics. Notably, our model provides both an overall score and individual scores for each evaluation item, enhancing interpretability. We also demonstrate its flexible training across various evaluation systems.

ReFINE: A Reward-Based Framework for Interpretable and Nuanced Evaluation of Radiology Report Generation

TL;DR

ReFINE tackles the misalignment between automated radiology report evaluation and human judgment by introducing a reward-based metric trained with an LLM reward model and a margin-based loss. It leverages GPT-4 to generate labeled training samples across two scoring frameworks (RadCliQ and MRScore) and fine-tunes a LoRA-enabled Llama3 to output per-criterion rewards that sum to an overall score. The approach yields higher alignment with radiologists than traditional metrics and offers interpretable sub-scores that pinpoint specific errors. Its offline deployment and criterion flexibility make ReFINE a scalable, privacy-conscious tool for evaluating radiology reports with practical clinical impact.

Abstract

Automated radiology report generation (R2Gen) has advanced significantly, introducing challenges in accurate evaluation due to its complexity. Traditional metrics often fall short by relying on rigid word-matching or focusing only on pathological entities, leading to inconsistencies with human assessments. To bridge this gap, we introduce ReFINE, an automatic evaluation metric designed specifically for R2Gen. Our metric utilizes a reward model, guided by our margin-based reward enforcement loss, along with a tailored training data design that enables customization of evaluation criteria to suit user-defined needs. It not only scores reports according to user-specified criteria but also provides detailed sub-scores, enhancing interpretability and allowing users to adjust the criteria between different aspects of reports. Leveraging GPT-4, we designed an easy-to-use data generation pipeline, enabling us to produce extensive training data based on two distinct scoring systems, each containing reports of varying quality along with corresponding scores. These GPT-generated reports are then paired as accepted and rejected samples through our pairing rule to train an LLM towards our fine-grained reward model, which assigns higher rewards to the report with high quality. Our reward-control loss enables this model to simultaneously output multiple individual rewards corresponding to the number of evaluation criteria, with their summation as our final ReFINE. Our experiments demonstrate ReFINE's heightened correlation with human judgments and superior performance in model selection compared to traditional metrics. Notably, our model provides both an overall score and individual scores for each evaluation item, enhancing interpretability. We also demonstrate its flexible training across various evaluation systems.

Paper Structure

This paper contains 17 sections, 1 equation, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation Metrics for Radiology Reports
  4. Large Language Model for Evaluation
  5. Method
  6. Generating Training Data by GPT-4
  7. LLM-based Reward Model
  8. Margin Reward Enforcement (MRE) Loss Function
  9. Experiments and Result
  10. Datasets
  11. Performance on ReXVal Dataset
  12. Performance on Rad-100 Dataest
  13. Performance Comparison of LLM backbones
  14. Ablation study of losses and hyperparameters
  15. Qualitative Analysis
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Overview of ReFINE. The upper portion illustrates the training data generation process, while the lower portion represents the training process for the reward model using LoRA. In the lower portion, the solid line indicates the training phase while the dashed line indicates the inference phase.
  • Figure 2: An illustration of report pairing rule, taking a scoring system with 4 criteria as an example.
  • Figure 3: An visual example of ReFINE from ReXVal Dataset. The highlighted sentences in reports and their corresponding scores share the same colors.