On the Empirical Power of Goodness-of-Fit Tests in Watermark Detection

TL;DR

This work investigates the applicability of classical goodness-of-fit (GoF) tests to watermark detection in large language models. By empirically evaluating eight GoF detectors across three unbiased watermarking schemes, three open-source LLMs, two datasets, multiple temperatures, and various edits, the authors show that GoF tests can improve detection power and robustness compared to baseline sum-based detectors. A key finding is that text repetition at low temperatures creates distributional structure that GoF tests exploit, while at high temperatures GoF tests leverage pronounced shifts in the empirical CDF of pivotal statistics. The study demonstrates that GoF-based watermark detection is simple, robust, and complementary to existing methods, and highlights future directions including theoretical analysis, adaptive test selection, and applications to higher-dimensional statistics. The work provides practical insights for deploying watermark detectors and underscores the value of distribution-level testing in NLP.

Abstract

Large language models (LLMs) raise concerns about content authenticity and integrity because they can generate human-like text at scale. Text watermarks, which embed detectable statistical signals into generated text, offer a provable way to verify content origin. Many detection methods rely on pivotal statistics that are i.i.d. under human-written text, making goodness-of-fit (GoF) tests a natural tool for watermark detection. However, GoF tests remain largely underexplored in this setting. In this paper, we systematically evaluate eight GoF tests across three popular watermarking schemes, using three open-source LLMs, two datasets, various generation temperatures, and multiple post-editing methods. We find that general GoF tests can improve both the detection power and robustness of watermark detectors. Notably, we observe that text repetition, common in low-temperature settings, gives GoF tests a unique advantage not exploited by existing methods. Our results highlight that classic GoF tests are a simple yet powerful and underused tool for watermark detection in LLMs.

Figures (9)

• Figure 1: Empirical CDFs of pivotal statistics ($F_0(Y_t)$) under different temperatures $\mathrm{T}$ and text lengths $n$. Shaded regions show 95% confidence intervals (CI): light gray for $n = 100$ and dark gray for $n = 400$, both at $\mathrm{T} = 1.0$. At $\mathrm{T} = 0.1$, the dark blue curve shows the empirical CDF of raw scores, while the orange curve (labeled $\mathrm{T} = 0.1^*$) shows the CDF after removing repeated values.
• Figure 2: Left: Average repetition rate in Gumbel-max watermarked texts across three LLMs, compared to human-written text. Right: Distribution of the highest probability in NTP distributions for OPT-1.3B using the Gumbel-max watermark at temperature 0.7.
• Figure 3: Change in Type II error after removing repeated pivotal statistics (averaged over three LLMs). Raw tokens denotes the error rate on the original 200-token sequence; Unique tokens denotes the error rate after excluding repeated pivotal statistics from the same sequence.
• Figure 4: Theoretical vs. empirical false positive rates (FPR) for GoF tests. Results are based on 100k sampled human-written texts from the C4 dataset, with repeated pivotals removed. Even at this scale, GoF tests maintain strong Type I error control, achieving empirical FPRs as low as the target significance level.
• Figure 5: Empirical Type II errors for various detection tests applied to the Gumbel-max watermark across the C4 dataset (top three rows) and the ELI5 dataset (bottom three rows).

Theorems & Definitions (2)

• Remark 4.1: Why we don't include green-red list watermark
• Remark 4.2: Differences from watermark stealing