Semantic Differentiation for Tackling Challenges in Watermarking Low-Entropy Constrained Generation Outputs

TL;DR

This work tackles watermarking for low-entropy, constrained generation by linking sequence-level entropy to watermark detectability and identifying region-collapse as a key failure mode in prior semantic watermarking approaches. It introduces SeqMark, a sequence-level watermarking method that isolates the high-quality output manifold and semantically differentiates its members before applying mean-centered, LSH-based partitioning to prevent collapse and evenly distribute desirable outputs across regions. The approach uses rejection sampling guided by transformed embeddings, and is supported by a theoretical bound (Theorem 1) showing mean-centering can improve watermark detectability. Empirically, SeqMark significantly improves detection accuracy (up to 28% F1 gains) across translation, summarization, and code generation while preserving generation quality, outperforming token-level baselines and prior sequence-level methods, particularly in constrained tasks. The work also introduces a Fast-SeqMark variant to reduce detection-time LM access, and discusses limitations such as detection-time LM access and potential paraphrase resilience, offering a practical, scalable improvement for watermarking constrained LM outputs.

Abstract

We demonstrate that while the current approaches for language model watermarking are effective for open-ended generation, they are inadequate at watermarking LM outputs for constrained generation tasks with low-entropy output spaces. Therefore, we devise SeqMark, a sequence-level watermarking algorithm with semantic differentiation that balances the output quality, watermark detectability, and imperceptibility. It improves on the shortcomings of the prevalent token-level watermarking algorithms that cause under-utilization of the sequence-level entropy available for constrained generation tasks. Moreover, we identify and improve upon a different failure mode we term region collapse, associated with prior sequence-level watermarking algorithms. This occurs because the pseudorandom partitioning of semantic space for watermarking in these approaches causes all high-probability outputs to collapse into either invalid or valid regions, leading to a trade-off in output quality and watermarking effectiveness. SeqMark instead, differentiates the high-probable output subspace and partitions it into valid and invalid regions, ensuring the even spread of high-quality outputs among all the regions. On various constrained generation tasks like machine translation, code generation, and abstractive summarization, SeqMark substantially improves watermark detection accuracy (up to 28% increase in F1) while maintaining high generation quality.

Figures (4)

• Figure 1: Pareto frontiers of token-level watermarking KGW (blue), and two sequence-level watermarking approaches SemStamp (orange) and SeqMark (ours) (green) for machine translation (Left) and summarization (Right).
• Figure 2: Semantic space illustration for different sequence watermarking algorithms. Both SemStamp (Left) and $k$-SemStamp (Middle) suffer from region collapse. Ideally, we would like to isolate the high-quality output manifold and partition it (Right).
• Figure 3: Depiction of SeqMark: Standard LSH in the first panel collapses the high-probable output subspace (blue) to a small number of invalid regions (red). We identify the high-probable subspace by sampling with low temperature; then we repartition by performing LSH on a transformed subspace designed to evenly spread it across multiple regions.
• Figure 4: Visual illustration of ROC curve for sentence translation watermark detection, using SeqMark as an example. Given there is only a single decision threshold, the AUROC is computed using only three points.