A Robust Semantics-based Watermark for Large Language Model against Paraphrasing

TL;DR

This work tackles the vulnerability of existing watermarking methods to paraphrase by shifting from token-hash based seeds to semantics-based seeds. It introduces SemaMark, which discretizes semantic embeddings on a Normalized Embedding Ring (NE-Ring) and uses contrastive learning to ensure uniform coverage of semantic values, coupled with Q-offset detection to handle boundary perturbations. Empirical results on multiple LLM backbones and paraphrase types show SemaMark achieves superior robustness to paraphrase with minimal impact on text quality, outperforming baseline watermarking methods. The approach enhances practical detectability of LLM-generated content in real-world deployments where paraphrase attacks are feasible, though it relies on access to embeddings and logits and may require LLM-specific adaptations.

Abstract

Large language models (LLMs) have show great ability in various natural language tasks. However, there are concerns that LLMs are possible to be used improperly or even illegally. To prevent the malicious usage of LLMs, detecting LLM-generated text becomes crucial in the deployment of LLM applications. Watermarking is an effective strategy to detect the LLM-generated content by encoding a pre-defined secret watermark to facilitate the detection process. However, the majority of existing watermark methods leverage the simple hashes of precedent tokens to partition vocabulary. Such watermark can be easily eliminated by paraphrase and correspondingly the detection effectiveness will be greatly compromised. Thus, to enhance the robustness against paraphrase, we propose a semantics-based watermark framework SemaMark. It leverages the semantics as an alternative to simple hashes of tokens since the paraphrase will likely preserve the semantic meaning of the sentences. Comprehensive experiments are conducted to demonstrate the effectiveness and robustness of SemaMark under different paraphrases.

Figures (6)

• Figure 1: The watermarking process of SemaMark
• Figure 2: $Q$-offset detection vs. existing detection
• Figure 3: Text quality (perplexity)
• Figure 4: ROC-AUC and $m$
• Figure 5: Text quality (perplexity)