Improving Chest X-Ray Report Generation by Leveraging Warm Starting

TL;DR

This work studies warm starting for CXR report generation by evaluating a wide range of CV and NLP checkpoints for the encoder and decoder, respectively. The authors identify CvT-21 ImageNet-21K as the most effective encoder and DistilGPT2 as the strongest decoder, yielding superior diagnostic-aligned and linguistically aligned reports compared to the prior $ ext{$ ext{M}^2$ Transformer Progressive}$ baseline. They demonstrate substantial gains in CE F-1 (8.3%), BLEU-4 (1.8%), ROUGE-L (1.6%), and METEOR (1.0%), and provide a detailed analysis of per-observation performance, a case-study on cross-attention, and insights into data size requirements. The study concludes that domain-specific pretraining and Transformer-based CV encoders combined with efficient NLP decoders substantially improve automated CXR report generation, with implications for clinical deployment and radiology workflow efficiency.

Abstract

Automatically generating a report from a patient's Chest X-Rays (CXRs) is a promising solution to reducing clinical workload and improving patient care. However, current CXR report generators -- which are predominantly encoder-to-decoder models -- lack the diagnostic accuracy to be deployed in a clinical setting. To improve CXR report generation, we investigate warm starting the encoder and decoder with recent open-source computer vision and natural language processing checkpoints, such as the Vision Transformer (ViT) and PubMedBERT. To this end, each checkpoint is evaluated on the MIMIC-CXR and IU X-Ray datasets. Our experimental investigation demonstrates that the Convolutional vision Transformer (CvT) ImageNet-21K and the Distilled Generative Pre-trained Transformer 2 (DistilGPT2) checkpoints are best for warm starting the encoder and decoder, respectively. Compared to the state-of-the-art ($\mathcal{M}^2$ Transformer Progressive), CvT2DistilGPT2 attained an improvement of 8.3\% for CE F-1, 1.8\% for BLEU-4, 1.6\% for ROUGE-L, and 1.0\% for METEOR. The reports generated by CvT2DistilGPT2 have a higher similarity to radiologist reports than previous approaches. This indicates that leveraging warm starting improves CXR report generation. Code and checkpoints for CvT2DistilGPT2 are available at https://github.com/aehrc/cvt2distilgpt2.

Figures (11)

• Figure 1: Encoder-to-decoder model for CXR report generation. The encoder extracts visual features from the CXR. Following this, the decoder autoregressively generates each subword of the radiology report based on the previously generated subwords and the visual features from the encoder. The example is DICOM 02aa804e-bde0afdd-112c0b34-7bc16630-4e384014 from study 50414267 of patient 10000032 from MIMIC-CXR. [BOS] is the beginning-of-sentence special token.
• Figure 2: Non-proportional Venn diagram of the vocabulary of each natural language domain. As CXR reports are included in Intensive Care Unit (ICU) Electronic Health Records (EHRs), the vocabulary of CXR reports is a subset of the vocabulary of ICU EHRs. The vocabulary of ICU EHRs is a subset of the vocabulary of biomedical natural language, which is a subset of the vocabulary of the general domain. An example of a corpus belonging to each domain is also given.
• Figure 3: CXR report generation framework for (left) MIMIC-CXR and (right) IU X-Ray as in Chen_2020. [BOS] is the beginning-of-sentence special token.
• Figure 4: The number of parameters for each checkpoint.
• Figure 5: High-level view of the architecture of CvT-21 2 DistilGPT2. See Figure \ref{['fig:encoder_to_decoder']} for a high-level depiction of the CXR report generation process with an encoder-to-decoder model. Here, it is configured for MIMIC-CXR with the Chen_2020 splits. $Q$, $K$, and $V$ are the queries, keys, and values, respectively, for multi-head attention Vaswani_2017. * indicates that the linear layers for $Q$, $K$, and $V$ are replaced with the convolutional layers depicted below the multi-head attention module. [BOS] is the beginning-of-sentence special token. $N_l$ is the number of layers for each stage, where $N_l=1$, $N_l=4$, and $N_l=16$ for the first, second, and third stage, respectively. The head for DistilGPT2 is the same used for language modelling. Subwords produced by DistilGPT2 are separated by a vertical bar.