Protecting Copyright of Medical Pre-trained Language Models: Training-Free Backdoor Model Watermarking

TL;DR

This paper tackles the copyright protection of medical pre-trained language models (Med-PLMs) by introducing a training-free backdoor watermarking approach. It embeds a watermark by replacing selected low-frequency trigger word embeddings with transformed embeddings of paired medical terms in the word embedding layer, enabling black-box ownership verification across NER, RE, and QA without retraining. The method demonstrates high watermarking effectiveness (WACC > 80%) and strong fidelity, while remaining robust to model extraction, pruning, and merging attacks, and achieves embedding in about 10 seconds. The approach offers practical and scalable copyright protection for Med-PLMs deployed via MLaaS, with validated generalizability across multiple medical tasks and model architectures.

Abstract

With the advancement of intelligent healthcare, medical pre-trained language models (Med-PLMs) have emerged and demonstrated significant effectiveness in downstream medical tasks. While these models are valuable assets, they are vulnerable to misuse and theft, requiring copyright protection. However, existing watermarking methods for pre-trained language models (PLMs) cannot be directly applied to Med-PLMs due to domain-task mismatch and inefficient watermark embedding. To fill this gap, we propose the first training-free backdoor model watermarking for Med-PLMs. Our method employs low-frequency words as triggers, embedding the watermark by replacing their embeddings in the model's word embedding layer with those of specific medical terms. The watermarked Med-PLMs produce the same output for triggers as for the corresponding specified medical terms. We leverage this unique mapping to design tailored watermark extraction schemes for different downstream tasks, thereby addressing the challenge of domain-task mismatch in previous methods. Experiments demonstrate superior effectiveness of our watermarking method across medical downstream tasks. Moreover, the method exhibits robustness against model extraction, pruning, fusion-based backdoor removal attacks, while maintaining high efficiency with 10-second watermark embedding.

Figures (10)

• Figure 1: Process of developing, deploying and applying Med-PLMs to various downstream tasks with potential model theft risks.
• Figure 2: Framework of the proposed Med-PLMs watermarking method. Contains three stages: (1) Using identity information and private keys to select low-frequency terms from medical corpora as triggers paired with corresponding medical terms (Sec \ref{['triggerselect']}). (2) Embedding watermarks in the word embedding layer of Med-PLMs (Sec \ref{['watermarkembed']}). (3) Extracting watermarks from final models in three core medical downstream tasks (Sec \ref{['watermarkdetect']}).
• Figure 3: Robustness of watermarking methods against model extraction: model performance and WACC of different method watermarked BioBERT across different tasks (NER/RE/QA) with varying extraction epochs.
• Figure 4: Robustness of our watermarking method against model pruning: model performance and WACC of watermarked BioBERT across medical downstream tasks (NER/RE/QA) with varying sparsity ratios (POR and PLMmark results in Appendix E).
• Figure 5: L2-distance based token embedding similarity in watermarked BioBERT's word embedding layer (darker colors indicate higher similarity).