Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DMark: Order-Agnostic Watermarking for Diffusion Large Language Models

Linyu Wu, Linhao Zhong, Wenjie Qu, Yuexin Li, Yue Liu, Shengfang Zhai, Chunhua Shen, Jiaheng Zhang

TL;DR

DMark tackles the challenge of watermarking diffusion-based LLMs, whose non-sequential token finalization invalidates traditional AR watermarks. It introduces three strategies—Predictive, Bidirectional, and Predictive-Bidirectional—leveraging parallel logit predictions and bidirectional context to embed detectable watermarks across arbitrary generation orders. Empirical results show strong detection rates at a 1% false-positive rate (92.0-99.5%), outperforming naive adaptations (49.6-71.2%), with robust performance under common text manipulations. The work provides both practical watermarking tools for dLLMs and theoretical groundwork for watermarking non-sequential generative models, with guidance on parameter settings and implementation details for reproducibility.

Abstract

Diffusion large language models (dLLMs) offer faster generation than autoregressive models while maintaining comparable quality, but existing watermarking methods fail on them due to their non-sequential decoding. Unlike autoregressive models that generate tokens left-to-right, dLLMs can finalize tokens in arbitrary order, breaking the causal design underlying traditional watermarks. We present DMark, the first watermarking framework designed specifically for dLLMs. DMark introduces three complementary strategies to restore watermark detectability: predictive watermarking uses model-predicted tokens when actual context is unavailable; bidirectional watermarking exploits both forward and backward dependencies unique to diffusion decoding; and predictive-bidirectional watermarking combines both approaches to maximize detection strength. Experiments across multiple dLLMs show that DMark achieves 92.0-99.5% detection rates at 1% false positive rate while maintaining text quality, compared to only 49.6-71.2% for naive adaptations of existing methods. DMark also demonstrates robustness against text manipulations, establishing that effective watermarking is feasible for non-autoregressive language models.

DMark: Order-Agnostic Watermarking for Diffusion Large Language Models

TL;DR

DMark tackles the challenge of watermarking diffusion-based LLMs, whose non-sequential token finalization invalidates traditional AR watermarks. It introduces three strategies—Predictive, Bidirectional, and Predictive-Bidirectional—leveraging parallel logit predictions and bidirectional context to embed detectable watermarks across arbitrary generation orders. Empirical results show strong detection rates at a 1% false-positive rate (92.0-99.5%), outperforming naive adaptations (49.6-71.2%), with robust performance under common text manipulations. The work provides both practical watermarking tools for dLLMs and theoretical groundwork for watermarking non-sequential generative models, with guidance on parameter settings and implementation details for reproducibility.

Abstract

Diffusion large language models (dLLMs) offer faster generation than autoregressive models while maintaining comparable quality, but existing watermarking methods fail on them due to their non-sequential decoding. Unlike autoregressive models that generate tokens left-to-right, dLLMs can finalize tokens in arbitrary order, breaking the causal design underlying traditional watermarks. We present DMark, the first watermarking framework designed specifically for dLLMs. DMark introduces three complementary strategies to restore watermark detectability: predictive watermarking uses model-predicted tokens when actual context is unavailable; bidirectional watermarking exploits both forward and backward dependencies unique to diffusion decoding; and predictive-bidirectional watermarking combines both approaches to maximize detection strength. Experiments across multiple dLLMs show that DMark achieves 92.0-99.5% detection rates at 1% false positive rate while maintaining text quality, compared to only 49.6-71.2% for naive adaptations of existing methods. DMark also demonstrates robustness against text manipulations, establishing that effective watermarking is feasible for non-autoregressive language models.

Paper Structure

This paper contains 31 sections, 14 equations, 3 figures, 4 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Diffusion Large Language Models (dLLMs)
  4. Watermarking for Autoregressive Models
  5. Methods
  6. Direct Adaptation of KGW
  7. Predictive Watermarking
  8. Bidirectional Watermarking
  9. Predictive-Bidirectional Watermarking
  10. Experiments
  11. Setups
  12. Models and Datasets
  13. Implementation
  14. Watermark Methods Performance
  15. Detection Effectiveness Across Methods.
  16. ...and 16 more sections

Figures (3)

  • Figure 1: Illustration of four watermarking methods for dLLMs. Gray rectangles represent finalized tokens, purple rectangles represent unfinalized tokens, yellow rectangles represent unfinalized but predicted tokens, and blue rectangles represent tokens to be generated. (a) KGW watermarking applies watermarks only when preceding context exists. (b) Predictive watermarking uses predicted preceding tokens as context when actual tokens are unavailable. (c) Bidirectional watermarking leverages both forward green lists and backward green lists. (d) Predictive-bidirectional watermarking combines prediction with bidirectional green lists for maximum watermark signals.
  • Figure 2: Impact of generation length on watermark detection strength (z-score) for different watermark parameters. Results shown for Predictive-Bidirectional watermarking on LLaDA-Instruct-8B with ELI5 dataset and different green list ratios ($\gamma$). Higher z-scores indicate stronger watermark detection, while lower PPL indicates better text quality.
  • Figure 3: Quality-detectability trade-off for varying watermark configurations. Each point represents a different combination of bias strength $\delta$ and green list ratio $\gamma$.