Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sandcastles in the Storm: Revisiting the (Im)possibility of Strong Watermarking

Fabrice Y Harel-Canada, Boran Erol, Connor Choi, Jason Liu, Gary Jiarui Song, Nanyun Peng, Amit Sahai

TL;DR

The study empirically tests the plausibility of defeating text watermarks via random-walk perturbations, challenging the theoretical claim of inevitable watermark removal. It shows slow mixing (texts retain lineage after hundreds of edits) and imperfect quality guidance from LLM-based oracles, with automated attacks removing watermarks only 26.1% of the time and as low as 10.5% after human review. The Sandcastles benchmark reveals substantial gaps between human quality judgments and automated checks, undermining the feasibility of oracle-guided attacks. The findings suggest watermarking remains robust under real-world constraints, highlighting the need for stronger schemes and more realistic threat models. Practical impact includes guiding watermark design toward resilience against imperfect evaluation and emphasizing human-aligned quality metrics in defense strategies.

Abstract

Watermarking AI-generated text is critical for combating misuse. Yet recent theoretical work argues that any watermark can be erased via random walk attacks that perturb text while preserving quality. However, such attacks rely on two key assumptions: (1) rapid mixing (watermarks dissolve quickly under perturbations) and (2) reliable quality preservation (automated quality oracles perfectly guide edits). Through large-scale experiments and human-validated assessments, we find mixing is slow: 100% of perturbed texts retain traces of their origin after hundreds of edits, defying rapid mixing. Oracles falter, as state-of-the-art quality detectors misjudge edits (77% accuracy), compounding errors during attacks. Ultimately, attacks underperform: automated walks remove watermarks just 26% of the time -- dropping to 10% under human quality review. These findings challenge the inevitability of watermark removal. Instead, practical barriers -- slow mixing and imperfect quality control -- reveal watermarking to be far more robust than theoretical models suggest. The gap between idealized attacks and real-world feasibility underscores the need for stronger watermarking methods and more realistic attack models.

Sandcastles in the Storm: Revisiting the (Im)possibility of Strong Watermarking

TL;DR

The study empirically tests the plausibility of defeating text watermarks via random-walk perturbations, challenging the theoretical claim of inevitable watermark removal. It shows slow mixing (texts retain lineage after hundreds of edits) and imperfect quality guidance from LLM-based oracles, with automated attacks removing watermarks only 26.1% of the time and as low as 10.5% after human review. The Sandcastles benchmark reveals substantial gaps between human quality judgments and automated checks, undermining the feasibility of oracle-guided attacks. The findings suggest watermarking remains robust under real-world constraints, highlighting the need for stronger schemes and more realistic threat models. Practical impact includes guiding watermark design toward resilience against imperfect evaluation and emphasizing human-aligned quality metrics in defense strategies.

Abstract

Watermarking AI-generated text is critical for combating misuse. Yet recent theoretical work argues that any watermark can be erased via random walk attacks that perturb text while preserving quality. However, such attacks rely on two key assumptions: (1) rapid mixing (watermarks dissolve quickly under perturbations) and (2) reliable quality preservation (automated quality oracles perfectly guide edits). Through large-scale experiments and human-validated assessments, we find mixing is slow: 100% of perturbed texts retain traces of their origin after hundreds of edits, defying rapid mixing. Oracles falter, as state-of-the-art quality detectors misjudge edits (77% accuracy), compounding errors during attacks. Ultimately, attacks underperform: automated walks remove watermarks just 26% of the time -- dropping to 10% under human quality review. These findings challenge the inevitability of watermark removal. Instead, practical barriers -- slow mixing and imperfect quality control -- reveal watermarking to be far more robust than theoretical models suggest. The gap between idealized attacks and real-world feasibility underscores the need for stronger watermarking methods and more realistic attack models.
Paper Structure (42 sections, 1 theorem, 18 equations, 11 figures, 13 tables)

This paper contains 42 sections, 1 theorem, 18 equations, 11 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Evaluation Setup
  4. Watermarkers.
  5. Dataset.
  6. Automatic Quality Metrics.
  7. Perturbation Oracles
  8. Analysis
  9. RQ1. Can Stationary Distributions Be Reached Under Practical Constraints?
  10. Results
  11. RQ2. Are LLM-based quality oracles sophisticated enough to guide a random-walk attack?
  12. The Sandcastles Benchmark.
  13. Constructing and Evaluating Oracles.
  14. Results
  15. Extended Comparison.
  16. ...and 27 more sections

Key Result

Theorem 1

Let $\Pi = (\text{Watermark}, \text{Detect})$ be a watermarking scheme for a class of generative models $\mathcal{M} = \{\mathbf{M}\xspace_i : \mathcal{X} \to \mathcal{Y}\}$ with an associated quality function $\mathbf{Q}\xspace : \mathcal{X} \times \mathcal{Y} \to [0,1]$. Let $\mathbf{P}\xspace : \ Assume the following holds: Then, there exists an oracle-aided universal adversary $A^{\mathbf{P}\

Figures (11)

  • Figure 1: InternLM Quality Distribution by Watermarking Scheme $\mathbf{W}\xspace$
  • Figure 2: Perplexity Distribution by Watermarking Scheme $\mathbf{W}\xspace$
  • Figure 3: Unique Bigrams Distribution by Watermarking Scheme $\mathbf{W}\xspace$
  • Figure 4: Grammar Errors Distribution by Watermarking Scheme $\mathbf{W}\xspace$
  • Figure 5: Attack success rate (ASR) vs. detection threshold for the Adaptive watermarking scheme. Each curve represents a different perturbation oracle, with thresholds measured in standard deviations above the unwatermarked mean.
  • ...and 6 more figures

Theorems & Definitions (10)

  • Definition 2.1: wits, Definition 9
  • Definition A.1: $\epsilon_{\text{pert}}\xspace$-Preserving Perturbation Oracle, wits, Definition 6
  • Definition A.2: wits, Definition 8
  • Definition A.3
  • Definition A.4
  • Definition A.5: wits, Definition 7
  • Definition A.6: wits, Definition 3
  • Definition A.7: wits, Definition 4
  • Definition A.8: wits, Definition 5
  • Theorem 1: wits, Theorem 6