dgMARK: Decoding-Guided Watermarking for Diffusion Language Models

TL;DR

dgMARK addresses provenance for discrete diffusion language models by leveraging decoding-order as a watermarking channel. It embeds signals through a parity-based hashing rule that guides the unmasking order, avoiding modifications to the model’s learned probabilities and remaining compatible with common decoding strategies. The approach includes a one-step lookahead variant and a robust sliding-window detector to maintain detection under post-editing, with empirical results showing strong detectability and minimal quality loss against baselines KGW and PATTERN-MARK. This decoding-guided method offers a practical, scalable provenance tool for dLLMs, robust to edits and suitable for long-form generation.

Abstract

We propose dgMARK, a decoding-guided watermarking method for discrete diffusion language models (dLLMs). Unlike autoregressive models, dLLMs can generate tokens in arbitrary order. While an ideal conditional predictor would be invariant to this order, practical dLLMs exhibit strong sensitivity to the unmasking order, creating a new channel for watermarking. dgMARK steers the unmasking order toward positions whose high-reward candidate tokens satisfy a simple parity constraint induced by a binary hash, without explicitly reweighting the model's learned probabilities. The method is plug-and-play with common decoding strategies (e.g., confidence, entropy, and margin-based ordering) and can be strengthened with a one-step lookahead variant. Watermarks are detected via elevated parity-matching statistics, and a sliding-window detector ensures robustness under post-editing operations including insertion, deletion, substitution, and paraphrasing.

Figures (17)

• Figure 1: Overview.(Left) Existing autoregressive watermarking methods generate green/red token sets by hashing the preceding context and embed watermark signals by biasing the sampling distribution toward green tokens. (Middle) In contrast, decoding in dLLMs does not follow the traditional left-to-right generation process; instead, the model selects high-reward tokens at each position even in the absence of prior context. (Right) The proposed method leverages these rewards and embeds watermark signals by prioritizing tokens with high reward that satisfy the parity condition.
• Figure 2: ROC curves under post-editing attacks. Illustration of the sliding-window strategy against random deletion, insertion, and substitution with modification budget $\epsilon$. Watermarks are generated by standard dgMARK ($k=1$) using multinomial sampling.
• Figure 3: Detection AUC under paraphrasing attacks. Results for dgMARK with DIPPER krishna2023paraphrasing: (Left) paraphrasing at predefined ratios via lexical modification; (Middle) paraphrasing with ratio-adjusted lexical modification and an additional 10$\%$ order diversity. Comparative results with KGW and PATTERN-MARK are included to assess relative robustness. (Right) paraphrasing generated by Llama 3-8B (Instruct), evaluated using ROC curves.
• Figure 4: Watermark detectability vs. sequence length. Results under multinomial sampling with beam sizes $k \in \{1,3,5,8\}$, reported as TPR at FPR levels of 10$\%$, 1$\%$, 0.1$\%$, and 0.01$\%$. Generation lengths are set to $\{16, 32, 64, 128, 256\}$, where the 256 setting includes sequences with length $\geq$ 200.
• Figure 5: Illustration of the distribution of parity alignment. At window size $w=32$, comparison of (1) non-watermarked texts (Non WM), (2) intact watermarked texts (WM), and (3) watermarked texts (WM) with "random token insertions", where the number of inserted tokens increases from left to right.