Advancing Beyond Identification: Multi-bit Watermark for Large Language Models

TL;DR

This work introduces Multi-bit Watermarking via Position Allocation (MPAC), a method that extends zero-bit watermarking to embed and reliably extract multi-bit information during large language model generation. By allocating each token to a specific message position and partitioning the vocabulary into colorlists, MPAC encodes long messages with minimal latency while preserving text quality and enabling zero-bit detection. The approach yields superior robustness to corruption (e.g., copy-paste, paraphrasing) and scales to 32+ bit messages through list decoding, without requiring finetuning. Empirical results on LLaMA-2-7B demonstrate strong performance across bit-widths, multiple datasets, and model variants, suggesting MPAC as a practical mechanism to trace adversaries and promote accountability in LLM API usage.

Abstract

We show the viability of tackling misuses of large language models beyond the identification of machine-generated text. While existing zero-bit watermark methods focus on detection only, some malicious misuses demand tracing the adversary user for counteracting them. To address this, we propose Multi-bit Watermark via Position Allocation, embedding traceable multi-bit information during language model generation. Through allocating tokens onto different parts of the messages, we embed longer messages in high corruption settings without added latency. By independently embedding sub-units of messages, the proposed method outperforms the existing works in terms of robustness and latency. Leveraging the benefits of zero-bit watermarking, our method enables robust extraction of the watermark without any model access, embedding and extraction of long messages ($\geq$ 32-bit) without finetuning, and maintaining text quality, while allowing zero-bit detection all at the same time. Code is released here: https://github.com/bangawayoo/mb-lm-watermarking

Figures (17)

• Figure 1: Comparison of how messages are encoded for zero-bit watermarking kirchenbauer2023watermark, recent multi-bit methods, and our proposed method MPAC. For MPAC, the number inside a token (e.g. $\boxed{p=1}$) denotes the allocated position.
• Figure 2: An overview of our method MPAC. See \ref{['subsec:multibit']} for details.
• Figure 3: Left: Comparison with prior works without corruption (clean) and in the presence of copy-paste attack with $p$%. On 24-bit, only 100 samples were watermarked for Cyclic-Shift and Message-Hash due to lengthened encoding / decoding time. Right: TPR for various FPR thresholds.
• Figure 4: Corrupted bit accuracy for paraphrasing attack using GPT-3.5 embedding 8-bit messages at varying token lengths. We show multiple sizes of list ($|L|\in${2, 4, 8, 16}) by color gradation as 8-bit has relatively small output space.
• Figure 5: