Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Overview of the First Shared Task on Clinical Text Generation: RRG24 and "Discharge Me!"

Justin Xu, Zhihong Chen, Andrew Johnston, Louis Blankemeier, Maya Varma, Jason Hom, William J. Collins, Ankit Modi, Robert Lloyd, Benjamin Hopkins, Curtis Langlotz, Jean-Benoit Delbrouck

TL;DR

The paper presents the first public shared task on clinical text generation with two subtasks: RRG24 for radiology report generation from chest X-rays and Discharge Me! for generating the Brief Hospital Course and Discharge Instructions in discharge summaries. It assembles Interpret-CXR as a large, multi-source dataset for RRG and uses Codabench for evaluation of Discharge Me!, reporting 201 and 211 submissions from eight and sixteen teams respectively. Evaluation combines standard automatic metrics (BLEU-4, ROUGE, BERTScore, F1-CheXbert, F1-RadGraph) and model-based measures, complemented by clinician reviews to assess completeness, correctness, and readability. Top systems leverage diverse multimodal large-language-model architectures with specialized prompts and adapters, demonstrating meaningful gains but also highlighting persistent challenges in factuality, clinical alignment, and workflow integration.

Abstract

Recent developments in natural language generation have tremendous implications for healthcare. For instance, state-of-the-art systems could automate the generation of sections in clinical reports to alleviate physician workload and streamline hospital documentation. To explore these applications, we present a shared task consisting of two subtasks: (1) Radiology Report Generation (RRG24) and (2) Discharge Summary Generation ("Discharge Me!"). RRG24 involves generating the 'Findings' and 'Impression' sections of radiology reports given chest X-rays. "Discharge Me!" involves generating the 'Brief Hospital Course' and 'Discharge Instructions' sections of discharge summaries for patients admitted through the emergency department. "Discharge Me!" submissions were subsequently reviewed by a team of clinicians. Both tasks emphasize the goal of reducing clinician burnout and repetitive workloads by generating documentation. We received 201 submissions from across 8 teams for RRG24, and 211 submissions from across 16 teams for "Discharge Me!".

Overview of the First Shared Task on Clinical Text Generation: RRG24 and "Discharge Me!"

TL;DR

The paper presents the first public shared task on clinical text generation with two subtasks: RRG24 for radiology report generation from chest X-rays and Discharge Me! for generating the Brief Hospital Course and Discharge Instructions in discharge summaries. It assembles Interpret-CXR as a large, multi-source dataset for RRG and uses Codabench for evaluation of Discharge Me!, reporting 201 and 211 submissions from eight and sixteen teams respectively. Evaluation combines standard automatic metrics (BLEU-4, ROUGE, BERTScore, F1-CheXbert, F1-RadGraph) and model-based measures, complemented by clinician reviews to assess completeness, correctness, and readability. Top systems leverage diverse multimodal large-language-model architectures with specialized prompts and adapters, demonstrating meaningful gains but also highlighting persistent challenges in factuality, clinical alignment, and workflow integration.

Abstract

Recent developments in natural language generation have tremendous implications for healthcare. For instance, state-of-the-art systems could automate the generation of sections in clinical reports to alleviate physician workload and streamline hospital documentation. To explore these applications, we present a shared task consisting of two subtasks: (1) Radiology Report Generation (RRG24) and (2) Discharge Summary Generation ("Discharge Me!"). RRG24 involves generating the 'Findings' and 'Impression' sections of radiology reports given chest X-rays. "Discharge Me!" involves generating the 'Brief Hospital Course' and 'Discharge Instructions' sections of discharge summaries for patients admitted through the emergency department. "Discharge Me!" submissions were subsequently reviewed by a team of clinicians. Both tasks emphasize the goal of reducing clinician burnout and repetitive workloads by generating documentation. We received 201 submissions from across 8 teams for RRG24, and 211 submissions from across 16 teams for "Discharge Me!".
Paper Structure (36 sections, 3 equations, 1 figure, 8 tables)

This paper contains 36 sections, 3 equations, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Radiology Report Generation
  4. Discharge Summary Generation
  5. RRG24: Radiology Report Generation
  6. Data
  7. Evaluation
  8. Results
  9. Descriptions of Systems
  10. e-Health CSIRO
  11. MAIRA
  12. AIRI
  13. Gla-AI4BioMed
  14. SICAR
  15. CID
  16. ...and 21 more sections

Figures (1)

  • Figure 1: Correlation heatmap visualizing interobserver agreement between clinician reviews. Cohen's Kappa scores were computed between pairwise clinicians based on the respective common submission(s) reviewed.