SSL-WM: A Black-Box Watermarking Approach for Encoders Pre-trained by Self-supervised Learning

TL;DR

SSL-WM introduces a black-box watermarking framework for self-supervised learning encoders that does not assume knowledge of downstream tasks. It embeds watermarks by shaping the encoder’s representation with a contrastive objective so watermarked inputs map to an invariant space, enabling downstream classifiers to produce predictable outputs for ownership verification via an entropy-based outlier test. The method applies to both contrastive-based and generative-based SSL models and demonstrates strong effectiveness, robustness to fine-tuning and pruning, and stealthiness against multiple watermark-detection approaches, outperforming the prior SSLGuard approach on several downstream tasks. This approach offers practical IP protection for SSL-encoded models deployed in diverse, unknown downstream contexts, with broad applicability to CV and NLP models and datasets.

Abstract

Recent years have witnessed tremendous success in Self-Supervised Learning (SSL), which has been widely utilized to facilitate various downstream tasks in Computer Vision (CV) and Natural Language Processing (NLP) domains. However, attackers may steal such SSL models and commercialize them for profit, making it crucial to verify the ownership of the SSL models. Most existing ownership protection solutions (e.g., backdoor-based watermarks) are designed for supervised learning models and cannot be used directly since they require that the models' downstream tasks and target labels be known and available during watermark embedding, which is not always possible in the domain of SSL. To address such a problem, especially when downstream tasks are diverse and unknown during watermark embedding, we propose a novel black-box watermarking solution, named SSL-WM, for verifying the ownership of SSL models. SSL-WM maps watermarked inputs of the protected encoders into an invariant representation space, which causes any downstream classifier to produce expected behavior, thus allowing the detection of embedded watermarks. We evaluate SSL-WM on numerous tasks, such as CV and NLP, using different SSL models both contrastive-based and generative-based. Experimental results demonstrate that SSL-WM can effectively verify the ownership of stolen SSL models in various downstream tasks. Furthermore, SSL-WM is robust against model fine-tuning, pruning, and input preprocessing attacks. Lastly, SSL-WM can also evade detection from evaluated watermark detection approaches, demonstrating its promising application in protecting the ownership of SSL models.

Figures (6)

• Figure 2: Visualization of embedding space of watermarked/clean Encoder generated by SimCLR in the downstream tasks using using T-SNE hinton2002stochastic. In these figures, the red cluster represents the embedding representation vectors of the watermarked input samples, and the other color clusters are the clean input samples, one color represents one class of samples in the downstream tasks. Moreover, there are 43 categories in the GTSRB dataset, it is difficult for us to visualize 43 colors to represent these 43 categories of clean data. Therefore, we uniformly represent these 43 categories of clean data in yellow.
• Figure 3: Robustness against Fine-tuning ($\lambda=0.00001$) and Pruning. Red dotted line is the threshold of MAD to verify ownership.
• Figure 4: Original watermark pattern and the reversed triggers by Neural Cleanse (NC) and ABS.
• Figure 5: Robustness against Fine-tuning ($\lambda=0.0001$). The red dotted line is the threshold of MAD to verify ownership.
• Figure 6: Visualization of embedding space of watermarked/clean self-supervised models using T-SNE hinton2002stochastic.