A Reinforcement Learning Framework for Robust and Secure LLM Watermarking

TL;DR

This work tackles robust and secure LLM watermarking by formulating green/red token-list design as an end-to-end reinforcement learning problem. It builds on semantic-aware watermarking and trains a lightweight mapping model as the policy while keeping the backbone LLM frozen, enabling direct optimization of detectability, robustness, and security. The authors introduce an anchored reward mechanism to stabilize training and an anti-reward-hacking regularization to prevent degenerate strategies, achieving state-of-the-art trade-offs across multiple criteria on two benchmarks and backbones. The framework demonstrates improved resistance to paraphrase and spoofing attacks with comparable text quality, and provides code for reproducibility and broader applicability with potential impact on provenance and content authentication. Limitations include the need to retrain the mapping model for new backbones, omission of explicit text-quality metrics beyond perplexity, and evaluation over a narrow set of attack types.

Abstract

Watermarking has emerged as a promising solution for tracing and authenticating text generated by large language models (LLMs). A common approach to LLM watermarking is to construct a green/red token list and assign higher or lower generation probabilities to the corresponding tokens, respectively. However, most existing watermarking algorithms rely on heuristic green/red token list designs, as directly optimizing the list design with techniques such as reinforcement learning (RL) comes with several challenges. First, desirable watermarking involves multiple criteria, i.e., detectability, text quality, robustness against removal attacks, and security against spoofing attacks. Directly optimizing for these criteria introduces many partially conflicting reward terms, leading to an unstable convergence process. Second, the vast action space of green/red token list choices is susceptible to reward hacking. In this paper, we propose an end-to-end RL framework for robust and secure LLM watermarking. Our approach adopts an anchoring mechanism for reward terms to ensure stable training and introduces additional regularization terms to prevent reward hacking. Experiments on standard benchmarks with two backbone LLMs show that our method achieves a state-of-the-art trade-off across all criteria, with notable improvements in resistance to spoofing attacks without degrading other criteria. Our code is available at https://github.com/UCSB-NLP-Chang/RL-watermark.

Figures (6)

• Figure 1: Performance comparison (higher is better) on detectability, robustness against paraphrase attacks, and security against sentiment and hate speech spoofing attacks. For detectability and paraphrase attack, AUCs are reported. For sentiment and hate attacks, scores are calculated using the complements (i.e., 100-AUC) of the corresponding AUCs.
• Figure 2: The RL training framework of our method. For each training instance, we perform the following three steps: ① Given an unwatermarked text $\bm x^{\text{uw}}$, sample and construct the green/red token assignment $\bm g$, and generate a watermarked text $\bm x^{\text{wm}}$. ② Compute the reward of $\bm g$ by evaluating the detection score on $\bm x^{\text{wm}}$ and its attacked variants, as well as on unwatermarked counterparts. ③ Update the mapping model to maximize the expected reward using GRPO, while keeping the backbone LLM frozen.
• Figure 3: Effect of anti-reward-hacking regularization on AUCs of attacked unwatermarked text.
• Figure 4: Prompt used for LLM as sentiment judge.
• Figure 5: Prompt used for sentiment spoofing attack.