WaterJudge: Quality-Detection Trade-off when Watermarking Large Language Models

TL;DR

WaterJudge presents a framework to quantify the quality-detection trade-off when watermarking large language models. It combines a soft-watermarking scheme with a zero-shot comparative assessment to map how watermark parameters affect both detection success and output quality, yielding actionable operating points. Across summarization and translation, the approach reveals predictable degradation patterns, high alignment with task-specific evaluators (e.g., UniEval, COMET), and promising transferability of settings between models and tasks. The work offers a practical, model- and task-agnostic tool for balancing watermark detectability with generation quality, with implications for deployment, evaluation, and policy.

Abstract

Watermarking generative-AI systems, such as LLMs, has gained considerable interest, driven by their enhanced capabilities across a wide range of tasks. Although current approaches have demonstrated that small, context-dependent shifts in the word distributions can be used to apply and detect watermarks, there has been little work in analyzing the impact that these perturbations have on the quality of generated texts. Balancing high detectability with minimal performance degradation is crucial in terms of selecting the appropriate watermarking setting; therefore this paper proposes a simple analysis framework where comparative assessment, a flexible NLG evaluation framework, is used to assess the quality degradation caused by a particular watermark setting. We demonstrate that our framework provides easy visualization of the quality-detection trade-off of watermark settings, enabling a simple solution to find an LLM watermark operating point that provides a well-balanced performance. This approach is applied to two different summarization systems and a translation system, enabling cross-model analysis for a task, and cross-task analysis.

Figures (18)

• Figure 1: High-level overview of the WaterJudge Framework: Given a system, watermarking parameters, and set of inputs, watermarked outputs are assessed in terms of quality and detectability, leading to a curve over all operating points.
• Figure 2: Comparative assessment probabilities are attained by calculating the likelihood of generating 'Text A' or 'Text B', normalizing, and averaging over both permutations. Example prompts are displayed, with the actual prompts shown in Appendix \ref{['sec:appendix_prompts']}.
• Figure 3: The trade-off between quality and detectability when watermarking. Each point is a watermark setting with green list size $g$ and bias $\delta$, displaying $F_{0.5}$ detectability score and average Comparative Assessment probability.
• Figure 4: Results for watermarked translations generated with mBART for combinations of green list size $g$ and bias $\delta$.
• Figure 5: Scatter plot showing correlation between Comparative Assessment and UniEval for BART.