Waterfall: Framework for Robust and Scalable Text Watermarking and Provenance for LLMs

TL;DR

It is demonstrated that Waterfall achieves significantly better scalability, robust verifiability, and computational efficiency compared to SOTA article-text watermarking methods, and showed how it could be directly applied to the watermarking of code.

Abstract

Protecting intellectual property (IP) of text such as articles and code is increasingly important, especially as sophisticated attacks become possible, such as paraphrasing by large language models (LLMs) or even unauthorized training of LLMs on copyrighted text to infringe such IP. However, existing text watermarking methods are not robust enough against such attacks nor scalable to millions of users for practical implementation. In this paper, we propose Waterfall, the first training-free framework for robust and scalable text watermarking applicable across multiple text types (e.g., articles, code) and languages supportable by LLMs, for general text and LLM data provenance. Waterfall comprises several key innovations, such as being the first to use LLM as paraphrasers for watermarking along with a novel combination of techniques that are surprisingly effective in achieving robust verifiability and scalability. We empirically demonstrate that Waterfall achieves significantly better scalability, robust verifiability, and computational efficiency compared to SOTA article-text watermarking methods, and showed how it could be directly applied to the watermarking of code. We also demonstrated that Waterfall can be used for LLM data provenance, where the watermarks of LLM training data can be detected in LLM output, allowing for detection of unauthorized use of data for LLM training and potentially enabling model-centric watermarking of open-sourced LLMs which has been a limitation of existing LLM watermarking works. Our code is available at https://github.com/aoi3142/Waterfall.

Figures (24)

• Figure 1: Schematics of problem formulation. Client $i$ watermark text $T_{\text{o}}$ with ID $\mu_i$ to watermarked text $T_{\text{w}}^{(i)}$. After manipulation by a third party, client can verify watermark in $T_{\text{sus}}$.
• Figure 2: Intuition on permutation operators $\mathcal{P}$, $\mathcal{P}^{-1}$ applied on LLM logits $L$ and watermarking signal $G$ with toy example, Vec. (a)$\mathcal{P}$ applied to $L$ in the $V_o$ space results in 6 possible permutations in $V_w$ space. This averages to constant vector $\bar{L}$. (b) Similarly, $\mathcal{P}^{-1}$ applied to $G$ in $V_w$ produces permutations in $V_o$. These averages to constant vector $\bar{G}$. (c) With $k_{\pi}$ sampled uniformly from the possible keys $K_\pi$ over multiple LLM generation steps, $L+G$ in shows less distortion to $G$ in $V_w$ space, and to $L$ in $V_o$ space.
• Figure 3: Left: Watermarking schematic. ① LLM paraphraser takes in $T_o$, produces initial logits. ② $k_{\pi}$ and $k_{p}$ from ID $\mu$ and metadata $k_{p}$ for vocab permutation and perturbation function. ③ Perturb logits with \ref{['eq:perturbed_logits']}. ④ Sample perturbed logits, feed past tokens to the next iteration. Right: Verification schematic. ① Permute tokens from $T_{\text{sus}}$ into $V_w$ with $\mu$ and preceding $n-1$ tokens, to get average cumulative distribution. ② Compute perturbation function $\mathcal{F}_1(k_p)$ linked to $\mu$. ③ Compute verification score as inner product of $\mathcal{F}_1(k_p)$ and cumulative distribution, and compare with threshold.
• Figure 4: Higher watermarking strength $\kappa$ improves verifiability and extraction accuracy. (a) Increasing $\kappa$ trades off fidelity for higher verifiability. (b) Waterfall performs significantly better than benchmarks in text quality, achieving a text quality-verifiabilty Pareto frontier that is much higher benchmarks (plotted as single points since they do not have adjustable settings to balance the quality-verifiability trade-off).
• Figure 5: (a) Longer token length $N$ improves verifiability. (b) Combining more pieces of text improves extraction accuracy towards $100\%$. Extraction accuracy is significantly higher than random guess accuracy of $0.003125\%$.