Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WatME: Towards Lossless Watermarking Through Lexical Redundancy

Liang Chen, Yatao Bian, Yang Deng, Deng Cai, Shuaiyi Li, Peilin Zhao, Kam-fai Wong

TL;DR

This paper tackles the problem of watermarking text generated by large language models without significantly degrading output quality or harming emergent cognitive abilities. It introduces WatME, a lossless watermarking approach that exploits lexical redundancy by forming interchangeable token clusters and enforcing mutual exclusivity across clusters during decoding. Theoretical analysis and extensive experiments show WatME preserves knowledge recall and logical reasoning capabilities much better than baseline watermarking methods, while maintaining comparable detection performance. The work has practical implications for regulatory compliance and accountability in AI systems, enabling more reliable attribution of machine-generated text with minimal impact on user experience.

Abstract

Text watermarking has emerged as a pivotal technique for identifying machine-generated text. However, existing methods often rely on arbitrary vocabulary partitioning during decoding to embed watermarks, which compromises the availability of suitable tokens and significantly degrades the quality of responses. This study assesses the impact of watermarking on different capabilities of large language models (LLMs) from a cognitive science lens. Our finding highlights a significant disparity; knowledge recall and logical reasoning are more adversely affected than language generation. These results suggest a more profound effect of watermarking on LLMs than previously understood. To address these challenges, we introduce Watermarking with Mutual Exclusion (WatME), a novel approach leveraging linguistic prior knowledge of inherent lexical redundancy in LLM vocabularies to seamlessly integrate watermarks. Specifically, WatME dynamically optimizes token usage during the decoding process by applying a mutually exclusive rule to the identified lexical redundancies. This strategy effectively prevents the unavailability of appropriate tokens and preserves the expressive power of LLMs. We provide both theoretical analysis and empirical evidence showing that WatME effectively preserves the diverse capabilities of LLMs while ensuring watermark detectability.

WatME: Towards Lossless Watermarking Through Lexical Redundancy

TL;DR

This paper tackles the problem of watermarking text generated by large language models without significantly degrading output quality or harming emergent cognitive abilities. It introduces WatME, a lossless watermarking approach that exploits lexical redundancy by forming interchangeable token clusters and enforcing mutual exclusivity across clusters during decoding. Theoretical analysis and extensive experiments show WatME preserves knowledge recall and logical reasoning capabilities much better than baseline watermarking methods, while maintaining comparable detection performance. The work has practical implications for regulatory compliance and accountability in AI systems, enabling more reliable attribution of machine-generated text with minimal impact on user experience.

Abstract

Text watermarking has emerged as a pivotal technique for identifying machine-generated text. However, existing methods often rely on arbitrary vocabulary partitioning during decoding to embed watermarks, which compromises the availability of suitable tokens and significantly degrades the quality of responses. This study assesses the impact of watermarking on different capabilities of large language models (LLMs) from a cognitive science lens. Our finding highlights a significant disparity; knowledge recall and logical reasoning are more adversely affected than language generation. These results suggest a more profound effect of watermarking on LLMs than previously understood. To address these challenges, we introduce Watermarking with Mutual Exclusion (WatME), a novel approach leveraging linguistic prior knowledge of inherent lexical redundancy in LLM vocabularies to seamlessly integrate watermarks. Specifically, WatME dynamically optimizes token usage during the decoding process by applying a mutually exclusive rule to the identified lexical redundancies. This strategy effectively prevents the unavailability of appropriate tokens and preserves the expressive power of LLMs. We provide both theoretical analysis and empirical evidence showing that WatME effectively preserves the diverse capabilities of LLMs while ensuring watermark detectability.
Paper Structure (41 sections, 2 theorems, 2 equations, 5 figures, 4 tables, 3 algorithms)

This paper contains 41 sections, 2 theorems, 2 equations, 5 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Preliminary
  5. Explore the Redundancy in Lexical Space
  6. Concept of Lexical Redundancy
  7. Constructing Redundant Lexical Clusters
  8. Handling Subword Tokenization
  9. Incorporating Grammatical Factors
  10. WatME: Exploit the Lexical Redundancy
  11. Theoretical Analysis
  12. Note:
  13. Impact on Emergent Abilities
  14. Knowledge Capability.
  15. Reasoning Capability.
  16. ...and 26 more sections

Key Result

Theorem 3.4

Consider that $\boldsymbol{p}_{t} \in \mathbb{R}^{|\mathcal{V}|}$ represents the predicted distribution of the model $\mathcal{M}$ at decoding time $t$. Let $w_i$ denote the token with the $i^{th}$ highest probability in $\boldsymbol{p}_{t}$. The higher the rank of a token (i.e., the smaller $i$ is)

Figures (5)

  • Figure 1: An illustration of WatME's advantage for lossless watermarking. The left panel depicts a vanilla LM with all words available during generation. The middle panel exposes the flaw in vanilla watermarking, which may assign all suitable tokens (e.g., 'ocean' and 'sea') to the red list, diminishing text quality. The right panel underlines how WatME exploits lexical redundancy by applying a mutual exclusion rule between such words, ensuring at least one suitable word remains on the green list, thereby improving text quality.
  • Figure 2: (a) Human evaluation for the quality of clusters mined by varied methods and (b) testing detection robustness against substitution attacks.
  • Figure 3: Performance trade-offs comparison between WatME and Vanilla Watermark on TruthfulQA and GSM8K at different Delta ($\Delta$) values.
  • Figure 4: Token-level entropy distributions for aligned (green) and unaligned (blue) models on GSM8K and TruthfulQA benchmarks.
  • Figure 5: Few-Shot Demonstration of Synonym Generation using LLMs.

Theorems & Definitions (4)

  • Definition 3.1: Semantic Entropy
  • Definition 3.2: Intrinsic Expressiveness
  • Theorem 3.4
  • Theorem 3.5