How to Train Private Clinical Language Models: A Comparative Study of Privacy-Preserving Pipelines for ICD-9 Coding

TL;DR

This study systematically compares four privacy-preserving pipelines for ICD-9 coding from hospital discharge summaries, using a fixed 1B-parameter model to isolate the effect of the privacy mechanism. Across privacy budgets $\varepsilon \in \{2,4,6\}$, knowledge distillation from DP-trained 3B teachers to a 1B student (DP-Distil) delivers the best utility under moderate to relaxed privacy while maintaining near-random membership-inference attack (MIA) privacy; direct DP-SGD (DP-Small) performs best only at the strictest budget, and DP-Synthetic lags due to low-fidelity synthetic data. Distillation even yields a 1B student that matches or exceeds its 3B teacher in utility, while reducing deployment costs. Synthetic data fails to capture essential clinical signals, and LoRA provides only partial implicit privacy, not a substitute for formal DP guarantees. Overall, DP knowledge distillation emerges as the most practical route to deployable, privacy-preserving clinical NLP on ICD-9 coding tasks, with clear guidance for healthcare AI deployments.

Abstract

Large language models trained on clinical text risk exposing sensitive patient information, yet differential privacy (DP) methods often severely degrade the diagnostic accuracy needed for deployment. Despite rapid progress in DP optimisation and text generation, it remains unclear which privacy-preserving strategy actually works best for clinical language tasks. We present the first systematic head-to-head comparison of four training pipelines for automated diagnostic coding from hospital discharge summaries. All pipelines use identical 1B-parameter models and matched privacy budgets to predict ICD-9 codes. At moderate and relaxed privacy budgets ($\varepsilon \in \{4, 6\}$), knowledge distillation from DP-trained teachers outperforms both direct DP-SGD and DP-synthetic data training, recovering up to 63\% of the non-private performance whilst maintaining strong empirical privacy (membership-inference AUC $\approx$ 0.5). These findings expose large differences in the privacy-utility trade-off across architectures and identify knowledge distillation as the most practical route to privacy-preserving clinical NLP.

Figures (5)

• Figure 1: Four training pipelines producing identical 1B classifiers (green). (A) DP-trained 3B generator creates synthetic data. (B) DP-trained 3B teachers (each $\varepsilon/2$) provide synthetic data and soft labels for knowledge distillation. (C) Direct DP-SGD on 1B model. (D) Non-private LoRA baseline. All DP pipelines use $\varepsilon \in \{2, 4, 6\}$ with $\delta = 10^{-5}$.
• Figure 2: Utility-privacy trade-off for Micro-F$_1$ (left) and Micro-AUPRC (right). DP-Distil and DP-Small improve monotonically with privacy budget, whilst DP-Synthetic plateaus regardless of $\varepsilon$. DP-Distil dominates at $\varepsilon \geq 4$; DP-Synthetic fails to scale.
• Figure 3: MIA vulnerability analysis. Left: Ensemble AUC showing LoRA-No-DP's higher vulnerability versus near-random performance for DP methods. Right: Individual feature contributions at $\varepsilon=4$, demonstrating consistent protection across attack vectors for DP methods. All DP methods achieve AUC $\approx$ 0.5; LoRA-No-DP remains vulnerable.
• Figure 4: Performance comparison between 3B-parameter DP-Distil teachers and their 1B-parameter distilled students across privacy budgets ($\varepsilon \in \{2, 4, 6\}$).
• Figure 5: Per-label F$_1$ scores for the ten most frequent ICD-9 codes, sorted by decreasing training frequency (n in parentheses). DP-Synthetic consistently underperforms DP-Small and DP-Distil across nearly all codes, while LoRA-No-DP shows irregular memorisation patterns.