Improving the Performance of Radiology Report De-identification with Large-Scale Training and Benchmarking Against Cloud Vendor Methods

TL;DR

This work tackles the challenge of radiology report PHI de-identification by scaling transformer-based models with large, multimodal training data and introducing an AGE PHI category. It shows that fine-tuning on Stanford CheXpert Plus and RadGraph-XL and evaluating with a hide-in-plain-sight synthetic PHI approach yields state-of-the-art or better PHI detection on real data (Penn and Stanford) and robust performance across synthetic benchmarks. The study also benchmarks against public cloud de-identification services using synthetic data, where the proposed model substantially outperforms competitors, highlighting limitations of current vendor solutions. Collectively, the results establish a new benchmark for privacy-preserving clinical text processing and demonstrate strong cross-institution generalization and privacy-utility tradeoffs that can facilitate broader, safer use of radiology text data for research and development.

Abstract

Objective: To enhance automated de-identification of radiology reports by scaling transformer-based models through extensive training datasets and benchmarking performance against commercial cloud vendor systems for protected health information (PHI) detection. Materials and Methods: In this retrospective study, we built upon a state-of-the-art, transformer-based, PHI de-identification pipeline by fine-tuning on two large annotated radiology corpora from Stanford University, encompassing chest X-ray, chest CT, abdomen/pelvis CT, and brain MR reports and introducing an additional PHI category (AGE) into the architecture. Model performance was evaluated on test sets from Stanford and the University of Pennsylvania (Penn) for token-level PHI detection. We further assessed (1) the stability of synthetic PHI generation using a "hide-in-plain-sight" method and (2) performance against commercial systems. Precision, recall, and F1 scores were computed across all PHI categories. Results: Our model achieved overall F1 scores of 0.973 on the Penn dataset and 0.996 on the Stanford dataset, outperforming or maintaining the previous state-of-the-art model performance. Synthetic PHI evaluation showed consistent detectability (overall F1: 0.959 [0.958-0.960]) across 50 independently de-identified Penn datasets. Our model outperformed all vendor systems on synthetic Penn reports (overall F1: 0.960 vs. 0.632-0.754). Discussion: Large-scale, multimodal training improved cross-institutional generalization and robustness. Synthetic PHI generation preserved data utility while ensuring privacy. Conclusion: A transformer-based de-identification model trained on diverse radiology datasets outperforms prior academic and commercial systems in PHI detection and establishes a new benchmark for secure clinical text processing.

Figures (2)

• Figure 1: Our pipeline for de-identifying protected health information (PHI) in clinical reports. Each original report is first passed through our transformer model to detect PHI across 8 different classes. Then, the PHI at the detected locations are replaced with realistic synthetic alternatives through the hide-in-plain-sight method. The final result is a de-identified report in which all PHI has been hidden with plausible synthetic substitutes.
• Figure 2: Overall summary of our model performance across real and synthetic data benchmarked against competitors.