Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representation

TL;DR

This work addresses the challenge of limited expert annotations in radiology text by introducing a two-stage framework that first uses LLMs to extract concise factual statements from chest X-ray reports and then encodes these facts with a fine-tuned BERT-based encoder (CXRFE) trained via multiple tasks to enhance factual representations. It introduces CXRFEScore, an embedding-based metric that assesses the factual alignment between reference and generated radiology text by comparing fact embeddings, defined as $CXRFEScore = \frac{S_{row}+S_{col}}{2}$ with $S_{row}=\frac{\sum_i \max_j M_{i,j}}{|R|}$ and $S_{col}=\frac{\sum_j \max_i M_{i,j}}{|C|}$, where $M$ is the cosine similarity matrix of fact embeddings. The approach leverages rich datasets (MIMIC-CXR, Chest ImaGenome, RadGraph, MedNLI, RadNLI, MS-CXR-T, IU X-ray) and extensive LLM-assisted data augmentation to train CXRFE across 11 task combinations, achieving state-of-the-art results on sentence ranking, NLI, and label extraction, while CXRFEScore outperforms traditional radiology-generation metrics across multiple benchmarks. Overall, the paper demonstrates that a carefully designed combination of fact extraction, multi-task encoding, and a fact-centric evaluation metric yields robust radiology text representations with practical implications for downstream clinical NLP tasks and radiology report evaluation.

Abstract

Advancing representation learning in specialized fields like medicine remains challenging due to the scarcity of expert annotations for text and images. To tackle this issue, we present a novel two-stage framework designed to extract high-quality factual statements from free-text radiology reports in order to improve the representations of text encoders and, consequently, their performance on various downstream tasks. In the first stage, we propose a \textit{Fact Extractor} that leverages large language models (LLMs) to identify factual statements from well-curated domain-specific datasets. In the second stage, we introduce a \textit{Fact Encoder} (CXRFE) based on a BERT model fine-tuned with objective functions designed to improve its representations using the extracted factual data. Our framework also includes a new embedding-based metric (CXRFEScore) for evaluating chest X-ray text generation systems, leveraging both stages of our approach. Extensive evaluations show that our fact extractor and encoder outperform current state-of-the-art methods in tasks such as sentence ranking, natural language inference, and label extraction from radiology reports. Additionally, our metric proves to be more robust and effective than existing metrics commonly used in the radiology report generation literature. The code of this project is available at \url{https://github.com/PabloMessina/CXR-Fact-Encoder}.

Figures (26)

• Figure 1: Example image and report from the IU X-ray dataset 10.1093/jamia/ocv080
• Figure 2: Fact extraction from radiology reports, designed to handle noisy input, including repeated words, incorrect sentence tokenization, typos, and verbose sentences, among other issues. When employing ChatGPT, context with instructions is used, whereas T5 is directly applied to the original sentence.
• Figure 3: Illustration of the computation of CXRFEScore. Given a reference report and a candidate report, we employ NLTK's sent_tokenize to extract sentences from each, extract facts from these sentences using T5, generate embeddings from the facts using CXRFE, and finally compute the pairwise cosine similarity matrix. Greedy matching is highlighted in red, with the calculation of the final score explicitly shown on the right. This figure's design is inspired by Figure 1 from BERTScore bert-score.
• Figure 4: Triplet loss (T)
• Figure 5: Sentence classification (C)