NSmark: Null Space Based Black-box Watermarking Defense Framework for Language Models

TL;DR

NSmark addresses the vulnerability of output-based watermarking for language models to LL-LFEA by exploiting null-space invariance in LM outputs. It introduces NSMD, a null-space matching metric, and a task-agnostic three-phase framework (generation, embedding, verification) that integrates spread-spectrum modulation and a mapping extractor to robustly embed watermarks while preserving model performance. The approach yields strong verification signals via Watermark Extraction Rate (WER) and NSMD, demonstrated across pre-training and downstream tasks and scalable to large LMs, with demonstrated resilience against LL-LFEA and related attacks. The work provides a practical, auditable defense for IP protection of LMs, and its open-source code facilitates real-world deployment and further research in robust black-box watermarking.

Abstract

Language models (LMs) have emerged as critical intellectual property (IP) assets that necessitate protection. Although various watermarking strategies have been proposed, they remain vulnerable to Linear Functionality Equivalence Attack (LFEA), which can invalidate most existing white-box watermarks without prior knowledge of the watermarking scheme or training data. This paper analyzes and extends the attack scenarios of LFEA to the commonly employed black-box settings for LMs by considering Last-Layer outputs (dubbed LL-LFEA). We discover that the null space of the output matrix remains invariant against LL-LFEA attacks. Based on this finding, we propose NSmark, a black-box watermarking scheme that is task-agnostic and capable of resisting LL-LFEA attacks. NSmark consists of three phases: (i) watermark generation using the digital signature of the owner, enhanced by spread spectrum modulation for increased robustness; (ii) watermark embedding through an output mapping extractor that preserves the LM performance while maximizing watermark capacity; (iii) watermark verification, assessed by extraction rate and null space conformity. Extensive experiments on both pre-training and downstream tasks confirm the effectiveness, scalability, reliability, fidelity, and robustness of our approach. Code is available at https://github.com/dongdongzhaoUP/NSmark.

Figures (9)

• Figure 1: Illustration of different watermark schemes against LFEA/LL-LFEA. LFEA disables parameters based white-box schemes li2023linear and LL-LFEA disables output based black-box schemes (Section \ref{['lfea']}). NSmark is secure against LL-LFEA using null space invariance.
• Figure 2: The schematic diagram of model inference flow before and after LL-LFEA attack. LL-LFEA transforms the LM output and performs an inverse transform in the subsequent linear layer, leaving the final prediction unchanged.
• Figure 3: The overall workflow of NSmark. (i) In watermark generation, identity information is used generate $sig$. (ii) In watermark embedding, watermarked model $f_{wm}$ and extractor $E$ are trained with the participation of the reference model $f_{ref}$. (iii) In watermark verification, WER and NSMD collaborate to verify the identity of the model.
• Figure 4: Example diagram of spread spectrum modulation. Repeat $sig$ to obtain $sig_{repeat}$, then use $sm$ to modulate $sig_{repeat}$ to obtain the spread spectrum modulated digital signature $sig_{sm}$.
• Figure 5: The impact of the correctness of trigger $t$ and signature $sig$ on WER and NSMD. The $c$ in the subscript stands for correct and $w$ stands for wrong. Only $f_{wm}$ with correct trigger and $sig$ can pass through verification.

Theorems & Definitions (3)

• Theorem 3.1
• proof
• proof