Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DSSmoothing: Toward Certified Dataset Ownership Verification for Pre-trained Language Models via Dual-Space Smoothing

Ting Qiao, Xing Liu, Wenke Huang, Jianbin Li, Zhaoxin Fan, Yiming Li

TL;DR

This work addresses the problem of unauthorized web-scale dataset usage in pre-trained language models by proposing the first certified dataset ownership verification (DOV) method for text, called DSSmoothing. It combines embedding-space continuous perturbations with permutation-space controlled reordering (dual-space smoothing) to embed robust watermarks and verify ownership with provable guarantees, expressed as $W(f_ heta, \uppi_{ ho, varepsilon}) > ext{threshold}$ under bounded perturbations. The authors provide a theoretical robustness theorem and validate the approach on AG's News and SST-2 across BERT, OPT-1.3B, and GPT-2, showing high watermark robustness (WSR ≈ 98%) and reliable ownership verification (low false positives) even under adaptive attacks and model transfer scenarios. The results demonstrate a practical, provably robust mechanism to protect dataset copyrights in large-scale language-model training, enabling trustworthy data sharing and governance.

Abstract

Large web-scale datasets have driven the rapid advancement of pre-trained language models (PLMs), but unauthorized data usage has raised serious copyright concerns. Existing dataset ownership verification (DOV) methods typically assume that watermarks remain stable during inference; however, this assumption often fails under natural noise and adversary-crafted perturbations. We propose the first certified dataset ownership verification method for PLMs based on dual-space smoothing (i.e., DSSmoothing). To address the challenges of text discreteness and semantic sensitivity, DSSmoothing introduces continuous perturbations in the embedding space to capture semantic robustness and applies controlled token reordering in the permutation space to capture sequential robustness. DSSmoothing consists of two stages: in the first stage, triggers are collaboratively embedded in both spaces to generate norm-constrained and robust watermarked datasets; in the second stage, randomized smoothing is applied in both spaces during verification to compute the watermark robustness (WR) of suspicious models and statistically compare it with the principal probability (PP) values of a set of benign models. Theoretically, DSSmoothing provides provable robustness guarantees for dataset ownership verification by ensuring that WR consistently exceeds PP under bounded dual-space perturbations. Extensive experiments on multiple representative web datasets demonstrate that DSSmoothing achieves stable and reliable verification performance and exhibits robustness against potential adaptive attacks.

DSSmoothing: Toward Certified Dataset Ownership Verification for Pre-trained Language Models via Dual-Space Smoothing

TL;DR

This work addresses the problem of unauthorized web-scale dataset usage in pre-trained language models by proposing the first certified dataset ownership verification (DOV) method for text, called DSSmoothing. It combines embedding-space continuous perturbations with permutation-space controlled reordering (dual-space smoothing) to embed robust watermarks and verify ownership with provable guarantees, expressed as under bounded perturbations. The authors provide a theoretical robustness theorem and validate the approach on AG's News and SST-2 across BERT, OPT-1.3B, and GPT-2, showing high watermark robustness (WSR ≈ 98%) and reliable ownership verification (low false positives) even under adaptive attacks and model transfer scenarios. The results demonstrate a practical, provably robust mechanism to protect dataset copyrights in large-scale language-model training, enabling trustworthy data sharing and governance.

Abstract

Large web-scale datasets have driven the rapid advancement of pre-trained language models (PLMs), but unauthorized data usage has raised serious copyright concerns. Existing dataset ownership verification (DOV) methods typically assume that watermarks remain stable during inference; however, this assumption often fails under natural noise and adversary-crafted perturbations. We propose the first certified dataset ownership verification method for PLMs based on dual-space smoothing (i.e., DSSmoothing). To address the challenges of text discreteness and semantic sensitivity, DSSmoothing introduces continuous perturbations in the embedding space to capture semantic robustness and applies controlled token reordering in the permutation space to capture sequential robustness. DSSmoothing consists of two stages: in the first stage, triggers are collaboratively embedded in both spaces to generate norm-constrained and robust watermarked datasets; in the second stage, randomized smoothing is applied in both spaces during verification to compute the watermark robustness (WR) of suspicious models and statistically compare it with the principal probability (PP) values of a set of benign models. Theoretically, DSSmoothing provides provable robustness guarantees for dataset ownership verification by ensuring that WR consistently exceeds PP under bounded dual-space perturbations. Extensive experiments on multiple representative web datasets demonstrate that DSSmoothing achieves stable and reliable verification performance and exhibits robustness against potential adaptive attacks.
Paper Structure (38 sections, 7 theorems, 17 equations, 14 figures, 2 tables)

This paper contains 38 sections, 7 theorems, 17 equations, 14 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background $\textbf{\&}$ Related Works
  3. Dataset Auditing
  4. Random Smoothing and CertDW
  5. Revisiting PLM Dataset Watermarking
  6. Vulnerability to Stochastic Noise
  7. Vulnerability under Adversary-Crafted Perturbations
  8. The Proposed Mrthod
  9. Preliminaries
  10. Threat Model.
  11. Overview of the Proposed Method
  12. Dataset Watermarking of DSSmoothing
  13. Ownership Verification of DSSmoothing
  14. Theoretical Analysis
  15. Experiments
  16. ...and 23 more sections

Key Result

Lemma 1

Let $(\bm{U},\bm{W})$ be the dual-space representation (see Definition def:dual-space-representation), with $\bm{U}\in\mathcal{U}$ a permutation matrix and $\bm{W}\in\mathcal{W}$ an embedding matrix, and let $h:\mathbb{R}^{n\times d}\to\mathcal{Y}$ be a base classifier. Let $\psi:\mathcal{U}\times\m Let $p_A=\mathbb{P}\![h(\psi(\bm{U},\bm{\rho})\cdot\phi(\bm{W},\bm{\varepsilon}))=y_A]$, $p_B=\max_

Figures (14)

  • Figure 1: Comparison between existing PLMs DOV methods and our proposed DSSmoothing. Existing DOV approaches are highly sensitive to natural noise and adversary-crafted perturbations, leading to degraded confidence and semantic corruption. In contrast, DSSmoothing applies dual-space smoothing in both embedding and permutation spaces to preserve semantics and enhance robustness, enabling certified and reliable ownership verification under bounded perturbations.
  • Figure 2: WSR of watermarked samples under perturbations in the embedding space. $d_N$ and $d_A$ denote the noise and adversary-crafted directions, respectively, and ‘$\bullet$’ marks the original watermarked sample. Left: BadWord; right: AddSent.
  • Figure 3: WSR of watermarked samples under noise.
  • Figure 4: The main pipeline of our DSSmoothing for certified PLMs dataset ownership verification.
  • Figure 5: Effects of the number of watermarked models.
  • ...and 9 more figures

Theorems & Definitions (9)

  • Definition 1: Embedding-Permutation Dual-Space Representation zhang2024text
  • Lemma 1: Certified Robustness in Dual-Space Randomized Smoothing zhang2024text
  • Lemma 1: Certified Robustness in Dual-Space Randomized Smoothing zhang2024text
  • Proposition 1: Ownership Verification via Dual-Space Conformal Prediction
  • Theorem 1: Robustness Condition of Certified Text Dataset Watermarking
  • Lemma 2: Lipschitz Continuity of Smoothed Classifier zhang2024text
  • Lemma 3
  • proof
  • Lemma 4: Certified Watermark Robustness in Permutation Space