UnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentation

TL;DR

This work proposes a novel Unlearnable Segmentation (UnSeg) framework to train a universal unlearnable noise generator that is capable of transforming any downstream images into their unlearnable version.

Abstract

Image segmentation is a crucial vision task that groups pixels within an image into semantically meaningful segments, which is pivotal in obtaining a fine-grained understanding of real-world scenes. However, an increasing privacy concern exists regarding training large-scale image segmentation models on unauthorized private data. In this work, we exploit the concept of unlearnable examples to make images unusable to model training by generating and adding unlearnable noise into the original images. Particularly, we propose a novel Unlearnable Segmentation (UnSeg) framework to train a universal unlearnable noise generator that is capable of transforming any downstream images into their unlearnable version. The unlearnable noise generator is finetuned from the Segment Anything Model (SAM) via bilevel optimization on an interactive segmentation dataset towards minimizing the training error of a surrogate model that shares the same architecture with SAM but is trained from scratch. We empirically verify the effectiveness of UnSeg across 6 mainstream image segmentation tasks, 10 widely used datasets, and 7 different network architectures, and show that the unlearnable images can reduce the segmentation performance by a large margin. Our work provides useful insights into how to leverage foundation models in a data-efficient and computationally affordable manner to protect images against image segmentation models.

Figures (17)

• Figure 1: An illustration of UnSeg pipeline which transforms images into unlearnable examples with mask prompt to prevent the exploitation of segmentation models.
• Figure 2: An overview of our proposed UnSeg framework. It finetunes an interactive unlearnable noise generator from the pre-trained SAM to generate unlearnable noise ($\delta^{u}$) that can minimize the training error of a surrogate model (a re-initialized SAM) via bilevel min-min optimization. After fine-tuning, only the unlearnable noise generator is kept.
• Figure 3: The training loss of UnSeg with/without EG and the validation results on Pascal VOC2012 using DeepLabV1 as target model.
• Figure 4: (a) The mIoU of DeepLabV1 trained on unlearnable Pascal VOC. (b) The mIoU of DeepLabV3 trained on unlearnable Pascal VOC. (3) The PQ of Mask2Former trained on unlearnable Cityscapes. The values were shown over different training epochs/iterations of the models.
• Figure 5: (a) The mIoU on 4 datasets of SAM-HQ ke2024segment trained on unlearnable HQSeg-44k ke2024segment. (b) The mAP on 3 datasets of RSPrompter chen2024rsprompter trained on their unlearnable training sets. (c) The IoU on 2 datasets candemir2013lungjha2020kvasir of UNet++ zhou2019unet++ trained on their unlearnable training sets with 5 different backbones.