Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CTest-Metric: A Unified Framework to Assess Clinical Validity of Metrics for CT Report Generation

Vanshali Sharma, Andrea Mia Bejar, Gorkem Durak, Ulas Bagci

TL;DR

The paper addresses the gap between conventional text-based metrics and clinical fidelity in CT radiology report generation. It introduces CTest-Metric, a three-module framework (WSG, SEI, MvE) to assess metric robustness and clinical relevance across seven LLM-based CT report pipelines using the CT-RATE dataset. Key findings show lexical NLG metrics are brittle to stylistic variation, while CE metrics vary in alignment with experts; GREEN Score shows strongest agreement with experts (approximately 0.70) and CRG may negatively correlate. The authors provide code, framework, and anonymized evaluation data to enable reproducible benchmarking and guide metric development.

Abstract

In the generative AI era, where even critical medical tasks are increasingly automated, radiology report generation (RRG) continues to rely on suboptimal metrics for quality assessment. Developing domain-specific metrics has therefore been an active area of research, yet it remains challenging due to the lack of a unified, well-defined framework to assess their robustness and applicability in clinical contexts. To address this, we present CTest-Metric, a first unified metric assessment framework with three modules determining the clinical feasibility of metrics for CT RRG. The modules test: (i) Writing Style Generalizability (WSG) via LLM-based rephrasing; (ii) Synthetic Error Injection (SEI) at graded severities; and (iii) Metrics-vs-Expert correlation (MvE) using clinician ratings on 175 "disagreement" cases. Eight widely used metrics (BLEU, ROUGE, METEOR, BERTScore-F1, F1-RadGraph, RaTEScore, GREEN Score, CRG) are studied across seven LLMs built on a CT-CLIP encoder. Using our novel framework, we found that lexical NLG metrics are highly sensitive to stylistic variations; GREEN Score aligns best with expert judgments (Spearman~0.70), while CRG shows negative correlation; and BERTScore-F1 is least sensitive to factual error injection. We will release the framework, code, and allowable portion of the anonymized evaluation data (rephrased/error-injected CT reports), to facilitate reproducible benchmarking and future metric development.

CTest-Metric: A Unified Framework to Assess Clinical Validity of Metrics for CT Report Generation

TL;DR

The paper addresses the gap between conventional text-based metrics and clinical fidelity in CT radiology report generation. It introduces CTest-Metric, a three-module framework (WSG, SEI, MvE) to assess metric robustness and clinical relevance across seven LLM-based CT report pipelines using the CT-RATE dataset. Key findings show lexical NLG metrics are brittle to stylistic variation, while CE metrics vary in alignment with experts; GREEN Score shows strongest agreement with experts (approximately 0.70) and CRG may negatively correlate. The authors provide code, framework, and anonymized evaluation data to enable reproducible benchmarking and guide metric development.

Abstract

In the generative AI era, where even critical medical tasks are increasingly automated, radiology report generation (RRG) continues to rely on suboptimal metrics for quality assessment. Developing domain-specific metrics has therefore been an active area of research, yet it remains challenging due to the lack of a unified, well-defined framework to assess their robustness and applicability in clinical contexts. To address this, we present CTest-Metric, a first unified metric assessment framework with three modules determining the clinical feasibility of metrics for CT RRG. The modules test: (i) Writing Style Generalizability (WSG) via LLM-based rephrasing; (ii) Synthetic Error Injection (SEI) at graded severities; and (iii) Metrics-vs-Expert correlation (MvE) using clinician ratings on 175 "disagreement" cases. Eight widely used metrics (BLEU, ROUGE, METEOR, BERTScore-F1, F1-RadGraph, RaTEScore, GREEN Score, CRG) are studied across seven LLMs built on a CT-CLIP encoder. Using our novel framework, we found that lexical NLG metrics are highly sensitive to stylistic variations; GREEN Score aligns best with expert judgments (Spearman~0.70), while CRG shows negative correlation; and BERTScore-F1 is least sensitive to factual error injection. We will release the framework, code, and allowable portion of the anonymized evaluation data (rephrased/error-injected CT reports), to facilitate reproducible benchmarking and future metric development.
Paper Structure (13 sections, 3 figures)

This paper contains 13 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Writing Style Generalizability Test (WSG)
  5. Synthetic Error Injection Test (SEI)
  6. Metrics-vs-Expert Correlation Test (MvE)
  7. Experiments and Results
  8. Dataset and Training Details
  9. Analyzing Writing Style Generalizability Test (WSG)
  10. Analyzing Synthetic Error Injection Test (SEI)
  11. Analyzing Metrics-vs-Expert Correlation Test (MvE)
  12. Discussion and Concluding Remarks
  13. Compliance with ethical standards

Figures (3)

  • Figure 1: (a) The propsoed framework, CTest-Metric, comprises three modules: (i) Writing Style Generalizability Test (WSG); (ii) Synthetic Error Injection Test (SEI); and (iii) Metrics‑vs‑Expert Correlation Test (MvE). (b) Sample reports are given.
  • Figure 2: Evaluation of metric reliability and robustness across rephrasing, factual error severity, and metric-expert correlations.
  • Figure 3: Metric response across increasing error levels.