Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GaussMark: A Practical Approach for Structural Watermarking of Language Models

Adam Block, Ayush Sekhari, Alexander Rakhlin

TL;DR

GaussMark introduces a practical, structure-aware watermark for language models by adding small Gaussian noise to a low-rank portion of a single MLP weight, producing a detectible signal without increasing generation latency. It casts watermark detection as a hypothesis test, proving the test is level-$\alpha$ with uniformly valid p-values and providing power bounds under linear-softmax and more general log-concavity assumptions. Empirically, GaussMark demonstrates strong detectability across multiple models, minimal impact on downstream task performance, and robustness to token-level corruptions and paraphrasing, with rank-reduced variants offering a favorable quality-detectability trade-off. The scheme offers a practical, white-box watermarking approach that can be integrated into existing inference pipelines, outperforming several prior methods in detection speed while maintaining text quality. Limitations include lack of distortion-freeness guarantees and the need for weight access, motivating future work on broader perturbation strategies and multi-watermark stacking to enhance robustness and applicability.

Abstract

Recent advances in Large Language Models (LLMs) have led to significant improvements in natural language processing tasks, but their ability to generate human-quality text raises significant ethical and operational concerns in settings where it is important to recognize whether or not a given text was generated by a human. Thus, recent work has focused on developing techniques for watermarking LLM-generated text, i.e., introducing an almost imperceptible signal that allows a provider equipped with a secret key to determine if given text was generated by their model. Current watermarking techniques are often not practical due to concerns with generation latency, detection time, degradation in text quality, or robustness. Many of these drawbacks come from the focus on token-level watermarking, which ignores the inherent structure of text. In this work, we introduce a new scheme, GaussMark, that is simple and efficient to implement, has formal statistical guarantees on its efficacy, comes at no cost in generation latency, and embeds the watermark into the weights of the model itself, providing a structural watermark. Our approach is based on Gaussian independence testing and is motivated by recent empirical observations that minor additive corruptions to LLM weights can result in models of identical (or even improved) quality. We show that by adding a small amount of Gaussian noise to the weights of a given LLM, we can watermark the model in a way that is statistically detectable by a provider who retains the secret key. We provide formal statistical bounds on the validity and power of our procedure. Through an extensive suite of experiments, we demonstrate that GaussMark is reliable, efficient, and relatively robust to corruptions such as insertions, deletions, substitutions, and roundtrip translations and can be instantiated with essentially no loss in model quality.

GaussMark: A Practical Approach for Structural Watermarking of Language Models

TL;DR

GaussMark introduces a practical, structure-aware watermark for language models by adding small Gaussian noise to a low-rank portion of a single MLP weight, producing a detectible signal without increasing generation latency. It casts watermark detection as a hypothesis test, proving the test is level- with uniformly valid p-values and providing power bounds under linear-softmax and more general log-concavity assumptions. Empirically, GaussMark demonstrates strong detectability across multiple models, minimal impact on downstream task performance, and robustness to token-level corruptions and paraphrasing, with rank-reduced variants offering a favorable quality-detectability trade-off. The scheme offers a practical, white-box watermarking approach that can be integrated into existing inference pipelines, outperforming several prior methods in detection speed while maintaining text quality. Limitations include lack of distortion-freeness guarantees and the need for weight access, motivating future work on broader perturbation strategies and multi-watermark stacking to enhance robustness and applicability.

Abstract

Recent advances in Large Language Models (LLMs) have led to significant improvements in natural language processing tasks, but their ability to generate human-quality text raises significant ethical and operational concerns in settings where it is important to recognize whether or not a given text was generated by a human. Thus, recent work has focused on developing techniques for watermarking LLM-generated text, i.e., introducing an almost imperceptible signal that allows a provider equipped with a secret key to determine if given text was generated by their model. Current watermarking techniques are often not practical due to concerns with generation latency, detection time, degradation in text quality, or robustness. Many of these drawbacks come from the focus on token-level watermarking, which ignores the inherent structure of text. In this work, we introduce a new scheme, GaussMark, that is simple and efficient to implement, has formal statistical guarantees on its efficacy, comes at no cost in generation latency, and embeds the watermark into the weights of the model itself, providing a structural watermark. Our approach is based on Gaussian independence testing and is motivated by recent empirical observations that minor additive corruptions to LLM weights can result in models of identical (or even improved) quality. We show that by adding a small amount of Gaussian noise to the weights of a given LLM, we can watermark the model in a way that is statistically detectable by a provider who retains the secret key. We provide formal statistical bounds on the validity and power of our procedure. Through an extensive suite of experiments, we demonstrate that GaussMark is reliable, efficient, and relatively robust to corruptions such as insertions, deletions, substitutions, and roundtrip translations and can be instantiated with essentially no loss in model quality.
Paper Structure (50 sections, 10 theorems, 58 equations, 29 figures, 13 tables, 4 algorithms)

This paper contains 50 sections, 10 theorems, 58 equations, 29 figures, 13 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Problem Setup and Prerequisites
  3. Hypothesis Testing
  4. Language Models
  5. Watermarking Text Generated by Language Models
  6. Additional Notation.
  7. Watermarking Scheme: $\mathsf{GaussMark}$
  8. Practical Implementation.
  9. Motivation
  10. Theoretical Analysis
  11. Linear Softmax Model
  12. Experiments
  13. Key Experimental Details.
  14. Can $\mathsf{GaussMark}$ be Detected?
  15. Does $\mathsf{GaussMark}$ Degrade Model Quality?
  16. ...and 35 more sections

Key Result

Proposition 3.1

Let $\{p_{\theta'}\}_{\theta' \in \mathbb{R}^d}$ be a family of measures on $\mathcal{Y}$, $x \in \mathcal{X}$, $\theta \in \mathbb{R}^d$, and $\nu \in \Delta(\mathbb{R}^d)$ be fixed. Suppose $\mathbf{H}_{\mathbf{0}} \vcentcolon={} \left\{\nu \otimes \mu \mid{} \mu \in \Delta(\mathcal{Y}) \right\}$

Figures (29)

  • Figure 1: Demonstration of the efficacy of GaussianMark on C4 prompts. (a) The median p-value of our detection procedure on 1K watermarked generations for different numbers of generated tokens averaged across 3 seeds. (b) Examples of watermarked text generated via $\mathsf{GaussMark}.\mathsf{Generate}$ on Llama3.1-8B model. The full text completions are given in \ref{['ssec:main_example']}
  • Figure 2: Effect of length of generated token sequence on p-values for $\mathsf{GaussMark}$ averaged over 3 seeds. (a) The fraction of detected watermarked responses at $p=0.05$ increases significantly as the number of tokens in the generated text increases, with $\mathsf{Phi3.5\textsf{--}Mini}$ being more challenging to watermark than $\mathsf{Llama}\mathsf{3.1}\textsf{--}\mathsf{8B}$ and $\mathsf{Mistral}\textsf{--}\mathsf{7B}$. (b) The increase of the gradient norm of the log-likelihood of the watermarked text with respect to the model parameters as the number of tokens in the generated text increases.
  • Figure 3: Latency of $\mathsf{GaussMark}$ in seconds for generation (a) and detection (b) as number of generated tokens increases. Note that both generation and detection processes are highly efficient, ensuring practical applicability of our approach.
  • Figure 4: Demonstration of robustness of $\mathsf{GaussMark}$ to four kinds of corruptions. We demonstrate the effects of (a) random insertions of tokens, (b) random deletions of tokens, and (c) random substitutions of tokens on the rate of detection of true positives of watermarked text averaged over 3 seeds. We also consider the effect that (d) roundtrip translation (through French) has on the ROC curve and include the ROC curves of the watermarked model on uncorrupted data for reference. Note that $\mathsf{GaussMark}$ is relatively robust to token-level corruptions and retains nontrivial power even after the more challenging roundtrip translation attack.
  • Figure 5: Demonstration of robustness of $\mathsf{GaussMark}$ to ignorance of prompt. We demonstrate the effects of (a) inserting, (b) deleting, and (c) substituting parts of the prompt on the fraction of detected sequences at the $p=0.05$ level. Note that $\mathsf{GaussMark}$ is robust to these corruptions, as the p-values remain relatively stable across different prompt corruptions, demonstrating that knowing the prompt is not necessary for watermark detection.
  • ...and 24 more figures

Theorems & Definitions (24)

  • Definition 2.1
  • Definition 2.2
  • Proposition 3.1
  • Proposition 3.2
  • Definition 3.3: Linear softmax model
  • Proposition 3.4
  • Corollary 3.5
  • Lemma G.1
  • proof
  • proof : Proof of prop:closest_measure
  • ...and 14 more