CSTRL: Context-Driven Sequential Transfer Learning for Abstractive Radiology Report Summarization

TL;DR

This work tackles automatic Impression generation from radiology Findings by proposing CSTRL, a context-driven sequential transfer learning framework. It first pre-trains on Gap Sentence Generation to extract pivotal content, then transfers to summarization using Fisher matrix regularization to mitigate catastrophic forgetting, complemented by contextual tagging and a teacher–student distillation mechanism. On MIMIC-CXR and Open-I, CSTRL achieves state-of-the-art BLEU and ROUGE scores and improves factual consistency, while delivering near-real-time inference with reduced computational requirements. The approach integrates clinical-context preservation with efficient deployment, offering a practical path toward reliable radiology report summarization in clinical workflows.

Abstract

A radiology report comprises several sections, including the Findings and Impression of the diagnosis. Automatically generating the Impression from the Findings is crucial for reducing radiologists' workload and improving diagnostic accuracy. Pretrained models that excel in common abstractive summarization problems encounter challenges when applied to specialized medical domains largely due to the complex terminology and the necessity for accurate clinical context. Such tasks in medical domains demand extracting core information, avoiding context shifts, and maintaining proper flow. Misuse of medical terms can lead to drastic clinical errors. To address these issues, we introduce a sequential transfer learning that ensures key content extraction and coherent summarization. Sequential transfer learning often faces challenges like initial parameter decay and knowledge loss, which we resolve with the Fisher matrix regularization. Using MIMIC-CXR and Open-I datasets, our model, CSTRL - Context-driven Sequential TRansfer Learning - achieved state-of-the-art performance, showing 56.2% improvement in BLEU-1, 40.5% in BLEU-2, 84.3% in BLEU-3, 28.9% in ROUGE-1, 41.0% in ROUGE-2 and 26.5% in ROGUE-3 score over benchmark studies. We also analyze factual consistency scores while preserving the medical context. Our code is publicly available at https://github.com/fahmidahossain/Report_Summarization.

Figures (5)

• Figure 1: Workflow of the proposed scheme for Sequential Transfer Learning. Firstly, sentences from the radiology corpus are scored with ROUGE and BLEU metrics, and top-scoring sentences are masked. Secondly, the T5 model is trained on the GSG task to predict the masked sentences. The Fisher Information matrix is calculated to identify critical parameters. Lastly, the penalty term is adjusted during fine-tuning for summarization.
• Figure 2: Workflow of the proposed scheme for contextual tagging. Firstly, keywords from the Impression section are extracted using TF-IDF vectorization. Secondly, relevant clinical terms are retrieved from the MRCONSO table, and tags are selected by searching keywords in the filtered table. Finally, the T5 model is trained to predict tags from the Findings section.
• Figure 3: Schematic representation of the proposed knowledge distillation network. The combined loss function is calculated using the reduced feature layers and the Hard Labels loss. Effective back-propagation ensures the optimization of the Student Model.
• Figure 4: Summary of CSTRL: The GSG Finetuned Teacher model (\ref{['fig_gsg_mask']}), using sequential transfer learning, is knowledge distilled, following the knowledge distillation process (\ref{['fig_kd_diagram']}). Additionally, a contextual tagging method (\ref{['fig:ner_diagram']}) is applied to the generated impressions.
• Figure 5: Fine-tuning with limited supervised examples. The solid lines are CSTRL fine-tuned on different numbers of datasets.