On the Learnability of Watermarks for Language Models

TL;DR

This work investigates whether language models can intrinsically learn to generate watermarked text through weights-based watermarking, bypassing specialized decoding. It introduces watermark distillation (logit-based and sampling-based) to transfer decoding-based signals (KGW, Aar, KTH) into the model's weights, enabling open-model watermarking. The study shows that high-distortion watermarks are learnable with strong detectability under standard decoding, but robustness to fine-tuning and potential spoofing attacks pose significant challenges. It also demonstrates a proof-of-concept spoofing attack, highlighting that watermarking should be used for statistical detection rather than attribution of provenance. Together, these results chart a path toward watermarking open models while underscoring practical security considerations and areas for further defense research.

Abstract

Watermarking of language model outputs enables statistical detection of model-generated text, which can mitigate harms and misuses of language models. Existing watermarking strategies operate by altering the decoder of an existing language model. In this paper, we ask whether language models can directly learn to generate watermarked text, which would have significant implications for the real-world deployment of watermarks. First, learned watermarks could be used to build open models that naturally generate watermarked text, enabling watermarking for open models, where users can control the decoding procedure. Second, if watermarking is used to determine the provenance of generated text, an adversary can hurt the reputation of a victim model by spoofing its watermark and generating damaging watermarked text. To investigate the learnability of watermarks, we propose watermark distillation, which trains a student model to behave like a teacher model that uses decoding-based watermarking. We test our approach on three decoding-based watermarking strategies and various hyperparameter settings, finding that models can learn to generate watermarked text with high detectability. We also find limitations to learnability, including the loss of watermarking capabilities under fine-tuning on normal text and high sample complexity when learning low-distortion watermarks.

Figures (5)

• Figure 1: Decoding-based watermarking (top) versus weights-based watermarking (bottom). Decoding-based watermarking requires a specialized decoding algorithm $f_\text{w}$ to generate watermarked text, whereas weights-based watermarking can use standard decoding to generate watermarked text directly from the model, using just its weights. Watermark distillation enables weights-based watermarking by training a student model $p_\theta$ to behave like the teacher model $p_{\mathrm{LM}}$ with decoding-based watermarking strategy $f_\text{w}$.
• Figure 2: Empirical cumulative distribution functions (eCDFs) of watermark detection p-values of generations from logit-based (a) and sampling-based (b) watermark distillation. In higher-distortion watermarks, the majority of generations have small p-values. In lower-distortion watermarks, the p-values are larger, but still consistently smaller than a non-watermarked uniform baseline.
• Figure 3: Watermark detection p-values of generations from weights-based watermarking, corrupted with varying proportions of tokens randomly edited. The watermarks are robust to moderate amounts of corruption.
• Figure 4: Watermark detection p-values of generations from logit-based watermark distilled Llama 2 7B models after further fine-tuning on OpenWebText. The models' weights-based watermarking is removed by fine-tuning.
• Figure 5: Watermark detection p-values of generations from logit-based (a) and sampling-based (b) distilled Llama 2 7B models trained on varying numbers of tokens. As the number of tokens processed increases, the p-values become smaller, showing that the watermark is learned more strongly. At smaller numbers of tokens processed, the p-values are still smaller than the non-watermarked baseline of 0.5, indicating that the watermark is still learned, albeit less strongly.