Synthetic Data for Veterinary EHR De-identification: Benefits, Limits, and Safety Trade-offs Under Fixed Compute

TL;DR

This study investigates the use of LLM-generated synthetic veterinary narratives to improve de-identification of vEHRs under resource constraints. It employs a privacy-preserving template-based generation approach and evaluates augmentation versus substitution across PetBERT, VetBERT, and Bio_ClinicalBERT, prioritizing document-level leakage as the safety metric. The results show that synthetic augmentation can reduce leakage and improve span-level performance when training exposure is increased (epoch-based), but synthetic data cannot safely substitute real labeled notes under fixed compute budgets; observed gains are largely exposure-driven and sensitive to data composition. Corpus diagnostics reveal systematic shifts in note length, label distribution, and repetition patterns between synthetic and real data, explaining leakage behavior and guiding future synthetic-data design. The findings underscore the need for compute-aware evaluation and careful handling of synthetic-real distribution shifts to safely deploy synthetic data for veterinary de-identification.

Abstract

Veterinary electronic health records (vEHRs) contain privacy-sensitive identifiers that limit secondary use. While PetEVAL provides a benchmark for veterinary de-identification, the domain remains low-resource. This study evaluates whether large language model (LLM)-generated synthetic narratives improve de-identification safety under distinct training regimes, emphasizing (i) synthetic augmentation and (ii) fixed-budget substitution. We conducted a controlled simulation using a PetEVAL-derived corpus (3,750 holdout/1,249 train). We generated 10,382 synthetic notes using a privacy-preserving "template-only" regime where identifiers were removed prior to LLM prompting. Three transformer backbones (PetBERT, VetBERT, Bio_ClinicalBERT) were trained under varying mixtures. Evaluation prioritized document-level leakage rate (the fraction of documents with at least one missed identifier) as the primary safety outcome. Results show that under fixed-sample substitution, replacing real notes with synthetic ones monotonically increased leakage, indicating synthetic data cannot safely replace real supervision. Under compute-matched training, moderate synthetic mixing matched real-only performance, but high synthetic dominance degraded utility. Conversely, epoch-scaled augmentation improved performance: PetBERT span-overlap F1 increased from 0.831 to 0.850 +/- 0.014, and leakage decreased from 6.32% to 4.02% +/- 0.19%. However, these gains largely reflect increased training exposure rather than intrinsic synthetic data quality. Corpus diagnostics revealed systematic synthetic-real mismatches in note length and label distribution that align with persistent leakage. We conclude that synthetic augmentation is effective for expanding exposure but is complementary, not substitutive, for safety-critical veterinary de-identification.

Figures (3)

• Figure 1: Synthetic augmentation sweep ($L{=}512$, stride $=64$; $n{=}3$ seeds). Points show mean; error bars show $\pm$1 SD across seeds. Top: Span-overlap F1 increases with synthetic fraction across backbones. Bottom: Document-level overlap leakage decreases with synthetic fraction, with PetBERT maintaining the lowest leakage across the sweep.
• Figure 2: Per-entity overlap recall across synthetic fractions. Synthetic augmentation drives recall gains in minority classes (e.g., LOC/ORG) while high-frequency classes (PER) change modestly.
• Figure 3: Training-regime sensitivity of synthetic augmentation (PetBERT). Left: Epoch-based training shows monotonic improvement in F1 and reduction in leakage. Right: Compute-matched training (fixed optimizer steps) shows performance peaks at moderate synthetic mixing (50%) rather than 90%.