Towards Codable Watermarking for Injecting Multi-bits Information to LLMs

TL;DR

This work tackles the need for multi-bit, codable watermarks in LLM-generated text, where prior methods offered only 1-bit signals. It formalizes Codable Text Watermarking (CTWL) as a encoding/decoding problem and introduces Balance-Marking, a pronunciation of CTWL that balances token-probability partitions using a proxy language model to preserve text quality while encoding information. The authors present two designs for the watermarking function $P_w$: Vanilla Pw (random vocabulary partition) and LM_proxy-aided Pw (balanced partition guided by a proxy LM), along with a Bayesian decoding framework and practical approximations. Extensive experiments across OPT-1.3B and LLaMA-7B/13B demonstrate Balance-Marking outperforms Vanilla-Marking in watermarking success, robustness to copy-paste and substitution attacks, and scalability to larger models, while highlighting trade-offs with encoding efficiency and proxy-LM size. The work lays the groundwork for open watermarking protocols and practical deployment of codable watermarks for provenance tracking and policy enforcement in AI-generated text.

Abstract

As large language models (LLMs) generate texts with increasing fluency and realism, there is a growing need to identify the source of texts to prevent the abuse of LLMs. Text watermarking techniques have proven reliable in distinguishing whether a text is generated by LLMs by injecting hidden patterns. However, we argue that existing LLM watermarking methods are encoding-inefficient and cannot flexibly meet the diverse information encoding needs (such as encoding model version, generation time, user id, etc.). In this work, we conduct the first systematic study on the topic of Codable Text Watermarking for LLMs (CTWL) that allows text watermarks to carry multi-bit customizable information. First of all, we study the taxonomy of LLM watermarking technologies and give a mathematical formulation for CTWL. Additionally, we provide a comprehensive evaluation system for CTWL: (1) watermarking success rate, (2) robustness against various corruptions, (3) coding rate of payload information, (4) encoding and decoding efficiency, (5) impacts on the quality of the generated text. To meet the requirements of these non-Pareto-improving metrics, we follow the most prominent vocabulary partition-based watermarking direction, and devise an advanced CTWL method named Balance-Marking. The core idea of our method is to use a proxy language model to split the vocabulary into probability-balanced parts, thereby effectively maintaining the quality of the watermarked text. Our code is available at https://github.com/lancopku/codable-watermarking-for-llm.

Figures (9)

• Figure 1: Balance-Marking outperforms Vanilla-marking under both different coding rates (subfigure a, b) and different attack scenarios (subfigure c, d).
• Figure 2: (a) Trade-off between efficiency and watermark success rate. Specifically, Vanilla-Marking can be viewed as Balance-Marking with $LM_{proxy} = \O$ (denoted as "null"). (b) Balance-Marking outperforms Vanilla-Marking in distinguishing between message-embedded and human-written texts. (c, d) Balance-Marking is still superior to Vanilla-Marking when using LLaMA-7/13B.
• Figure 3: Illustration of $LM_{proxy}$'s impact on watermarking. Larger $LM_{proxy}$ tends to exhibit improved performance.
• Figure 4: Illustration of $A$'s impact on watermarking quality. A higher $A$ tends to have better watermark quality.
• Figure 5: Illustration of $L_{prefix}$'s impact on watermark quality. A too-low $L_{prefix}$ value can degrade the watermark quality.