Ungeneralizable Examples

TL;DR

This work addresses data privacy concerns in training by allowing data to remain learnable for a predesignated authorized network while becoming unlearnable for potential hackers through Ungeneralizable Examples (UGEs). The authors introduce a triplet loss framework comprising gradient matching $L_{gm}$, feature distance $L_{fd}$, and undistill $L_{ud}$ losses, enabling a generator to produce ungeneralizable perturbations $x_u$ that preserve learning trajectories for the authorized model yet hinder unauthorized learning in shared or distillation-based settings. By leveraging a CLIP-based shared feature space and a misalignment strategy, UGEs maintain data usability for the authorized learner and resist various attacks, including distillation, across CIFAR-10/100 and TinyImageNet with diverse architectures. The approach is demonstrated in multiple usage scenarios, including decentralized training, safe code/data publication, and secure data transmission, and is extended to handle multiple authorized networks and federated learning contexts. Overall, UGEs offer a practical, extensible protection mechanism with robust empirical support and clear pathways for future enhancements in multi-network and downstream-task contexts.

Abstract

The training of contemporary deep learning models heavily relies on publicly available data, posing a risk of unauthorized access to online data and raising concerns about data privacy. Current approaches to creating unlearnable data involve incorporating small, specially designed noises, but these methods strictly limit data usability, overlooking its potential usage in authorized scenarios. In this paper, we extend the concept of unlearnable data to conditional data learnability and introduce \textbf{U}n\textbf{G}eneralizable \textbf{E}xamples (UGEs). UGEs exhibit learnability for authorized users while maintaining unlearnability for potential hackers. The protector defines the authorized network and optimizes UGEs to match the gradients of the original data and its ungeneralizable version, ensuring learnability. To prevent unauthorized learning, UGEs are trained by maximizing a designated distance loss in a common feature space. Additionally, to further safeguard the authorized side from potential attacks, we introduce additional undistillation optimization. Experimental results on multiple datasets and various networks demonstrate that the proposed UGEs framework preserves data usability while reducing training performance on hacker networks, even under different types of attacks.

Figures (7)

• Figure 1: The threat model of ungeneralizable examples involves generating UnGeneralizable Examples. Once created, both the protector and the hacker gain access to the UGEs rather than the original data. While the UGEs can effectively train the protector's network, they result in a performance drop on hacker networks.
• Figure 2: The comprehensive workflow of UGEs involves the protector training a generator to produce the ungeneralizable version of the original examples. Three distinct loss functions are employed in training the generator: gradient matching loss, feature distance loss, and undistill loss. Upon completion of the training process, the UGEs are published, and both the protector and hackers no longer have access to the original examples.
• Figure 3: The performance concerning the value of $\rho$ on CIFAR-10 and CIFAR-100 datasets.
• Figure 4: The visualization results include the original clean images, the ungeneralizable noise (scaled by $255$ for better visualization), and the resultant ungeneralizable images.
• Figure 5: The architecture of the generator to synthesize the ungeneralizable examples.