Standardizing Longitudinal Radiology Report Evaluation via Large Language Model Annotation

TL;DR

The paper tackles the lack of standardized tools for annotating longitudinal changes in radiology reports by introducing an LLM-based annotation pipeline that identifies longitudinal sentences and tracks disease progression. It builds L-MIMIC, a large annotated benchmark, from MIMIC-CXR to evaluate report-generation models on their ability to capture longitudinal information. The study shows that LLM-based annotation outperforms traditional methods (e.g., ImaGenome silver) in both detection and progression labeling, enabling rigorous, large-scale evaluation (e.g., $11.3\%$ higher $F1$ for detection and $5.3\%$ for progression). Seven state-of-the-art generation models are assessed, with Libra excelling on language metrics and Maira2 leading diagnostic/progression predictions, highlighting the remaining gaps in longitudinal information capture and guiding future improvements. Overall, the framework provides a practical, scalable benchmark for standardized longitudinal radiology report evaluation and model development.

Abstract

Longitudinal information in radiology reports refers to the sequential tracking of findings across multiple examinations over time, which is crucial for monitoring disease progression and guiding clinical decisions. Many recent automated radiology report generation methods are designed to capture longitudinal information; however, validating their performance is challenging. There is no proper tool to consistently label temporal changes in both ground-truth and model-generated texts for meaningful comparisons. Existing annotation methods are typically labor-intensive, relying on the use of manual lexicons and rules. Complex rules are closed-source, domain specific and hard to adapt, whereas overly simple ones tend to miss essential specialised information. Large language models (LLMs) offer a promising annotation alternative, as they are capable of capturing nuanced linguistic patterns and semantic similarities without extensive manual intervention. They also adapt well to new contexts. In this study, we therefore propose an LLM-based pipeline to automatically annotate longitudinal information in radiology reports. The pipeline first identifies sentences containing relevant information and then extracts the progression of diseases. We evaluate and compare five mainstream LLMs on these two tasks using 500 manually annotated reports. Considering both efficiency and performance, Qwen2.5-32B was subsequently selected and used to annotate another 95,169 reports from the public MIMIC-CXR dataset. Our Qwen2.5-32B-annotated dataset provided us with a standardized benchmark for evaluating report generation models. Using this new benchmark, we assessed seven state-of-the-art report generation models. Our LLM-based annotation method outperforms existing annotation solutions, achieving 11.3\% and 5.3\% higher F1-scores for longitudinal information detection and disease tracking, respectively.

Figures (4)

• Figure 1: The longitudinal information in the report and the advantages of LLM-based annotation. a. Take sentences from the second-visit report of a patient to illustrate cross-sectional sentences and longitudinal information in free-text reports. b. Comparing existing longitudinal information annotation methods with large model-based annotation methods.
• Figure 2: Annotation and evaluation pipeline. a. Longitudinal, keyword, and disease progression annotations. Longitudinal annotation identifies sentences containing longitudinal information; keyword annotation captures the main content; disease progression annotation captures changes in status (i.e., improved, no change, worsened, or unmentioned) related to the main content. b. An evaluation framework for the report generation model based on the proposed annotation method.
• Figure 3: Performance of large language models on longitudinal annotation and disease progression annotation. For disease progression annotation, the average refers to the micro-average across three classes: no change, improved, and worsened.
• Figure 4: Annotation statistics for follow-up reports in MIMIC-CXR. a. Proportion of sentences and reports with longitudinal information. L_report/L_sentence: reports/sentences with longitudinal information. C_report/C_sentence: reports/sentences that convey only cross-sectional information and exclude any longitudinal information. b. Proportion of disease progression categories in L-MIMIC.