Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Three Bricks to Consolidate Watermarks for Large Language Models

Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, Teddy Furon

TL;DR

This work tackles the problem of distinguishing generated from natural text by consolidating watermarking approaches for large language models. It introduces three bricks: (i) grounded, non-asymptotic statistical tests that guarantee false positive rates below $10^{-6}$, (ii) cross-benchmark evaluation to assess practical impact on downstream NLP tasks, and (iii) advanced detection schemes including Neyman–Pearson scoring and multi-bit watermarking for model/version tracing. The findings reveal that traditional $Z$-tests miscalibrate FPR in realistic regimes and that non-asymptotic statistics with rectified scoring provide reliable detection; watermarking has limited but manageable impact on generation quality, especially for larger models, and enables identification of the watermark source across many users. Collectively, the paper offers a practical, reliable framework for watermarking LLM outputs with implications for security, accountability, and model governance.

Abstract

The task of discerning between generated and natural texts is increasingly challenging. In this context, watermarking emerges as a promising technique for ascribing generated text to a specific model. It alters the sampling generation process so as to leave an invisible trace in the generated output, facilitating later detection. This research consolidates watermarks for large language models based on three theoretical and empirical considerations. First, we introduce new statistical tests that offer robust theoretical guarantees which remain valid even at low false-positive rates (less than 10$^{\text{-6}}$). Second, we compare the effectiveness of watermarks using classical benchmarks in the field of natural language processing, gaining insights into their real-world applicability. Third, we develop advanced detection schemes for scenarios where access to the LLM is available, as well as multi-bit watermarking.

Three Bricks to Consolidate Watermarks for Large Language Models

TL;DR

This work tackles the problem of distinguishing generated from natural text by consolidating watermarking approaches for large language models. It introduces three bricks: (i) grounded, non-asymptotic statistical tests that guarantee false positive rates below , (ii) cross-benchmark evaluation to assess practical impact on downstream NLP tasks, and (iii) advanced detection schemes including Neyman–Pearson scoring and multi-bit watermarking for model/version tracing. The findings reveal that traditional -tests miscalibrate FPR in realistic regimes and that non-asymptotic statistics with rectified scoring provide reliable detection; watermarking has limited but manageable impact on generation quality, especially for larger models, and enables identification of the watermark source across many users. Collectively, the paper offers a practical, reliable framework for watermarking LLM outputs with implications for security, accountability, and model governance.

Abstract

The task of discerning between generated and natural texts is increasingly challenging. In this context, watermarking emerges as a promising technique for ascribing generated text to a specific model. It alters the sampling generation process so as to leave an invisible trace in the generated output, facilitating later detection. This research consolidates watermarks for large language models based on three theoretical and empirical considerations. First, we introduce new statistical tests that offer robust theoretical guarantees which remain valid even at low false-positive rates (less than 10). Second, we compare the effectiveness of watermarks using classical benchmarks in the field of natural language processing, gaining insights into their real-world applicability. Third, we develop advanced detection schemes for scenarios where access to the LLM is available, as well as multi-bit watermarking.
Paper Structure (26 sections, 24 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 26 sections, 24 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Technical Background
  3. Large Language Models (LLMs)
  4. Watermarking Text Generation
  5. Modification of the Distribution kirchenbauer2023reliabilitykirchenbauer2023watermarkzhao2023provable
  6. Modification of the Sampling aaronson2023watermarkingchrist2023undetectable
  7. Quality-Robustness Trade-off
  8. Key Management
  9. $Z$-Test
  10. Reliability of the Detection
  11. Empirical Validation of FPR with Z-Scores
  12. New Non-Asymptotical Statistical Tests
  13. Kirchenbauer et al.
  14. Aaronson et al.
  15. Rectifying the Detection Scores
  16. ...and 11 more sections

Figures (3)

  • Figure 1: General illustration of watermarking for LLM (top: generation, bottom: detection). Details and notations in Sect. \ref{['sec:llm_wm']}.
  • Figure 2: Empirical checks of false positive rates for different watermarks and values of the context width $h$. Results are computed over $10$ master keys $\times$ 100k sequences of $256$ tokens sampled from Wikipedia. We compare three detection tests: (Left) using $Z$-tests; (Middle) using new statistical tests presented in \ref{['sec:new-stats']}; (Right) using the new statistical tests with the rectified scoring strategy of \ref{['sec:rect']}. Theoretical values do not hold in practice for $Z$-tests, even for high values of $h$, and empirical FPRs do not match theoretical ones. This is solved by basing detection on grounded statistical tests and analytic p-values, as well as by revising the scoring strategy.
  • Figure 3: Typical example of a vanilla text with low p-value because of repeated tokens. It is $10^{-21}$, using the greenlist watermark with $\gamma=1/4$ and $h=2$ on $256$ tokens (we only show half of the text).