Why Do Unlearnable Examples Work: A Novel Perspective of Mutual Information

TL;DR

This paper proposes a novel unlearnable method called Mutual Information Unlearnable Examples (MI-UE) that reduces covariance by maximizing the cosine similarity among intra-class features, thus impeding the generalization effectively.

Abstract

The volume of freely scraped data on the Internet has driven the tremendous success of deep learning. Along with this comes the growing concern about data privacy and security. Numerous methods for generating unlearnable examples have been proposed to prevent data from being illicitly learned by unauthorized deep models by impeding generalization. However, the existing approaches primarily rely on empirical heuristics, making it challenging to enhance unlearnable examples with solid explanations. In this paper, we analyze and improve unlearnable examples from a novel perspective: mutual information reduction. We demonstrate that effective unlearnable examples always decrease mutual information between clean features and poisoned features, and when the network gets deeper, the unlearnability goes better together with lower mutual information. Further, we prove from a covariance reduction perspective that minimizing the conditional covariance of intra-class poisoned features reduces the mutual information between distributions. Based on the theoretical results, we propose a novel unlearnable method called Mutual Information Unlearnable Examples (MI-UE) that reduces covariance by maximizing the cosine similarity among intra-class features, thus impeding the generalization effectively. Extensive experiments demonstrate that our approach significantly outperforms the previous methods, even under defense mechanisms.

Figures (10)

• Figure 1: The estimation of MI between clean and unlearnable poisoned features on different MI estimators. (a): MI metrics under histogram-based estimator and kernel density estimator (KDE). (b): MI metrics under histogram-based estimator and k-NN estimator. (c): MI metrics under histogram-based method and mutual information neural estimator (MINE). Green triangles represent clean or ineffective UEs, blue circles mean existing effective UEs, orange square denotes our MI-UE. It demonstrates that although different estimation methods show different quantitative results, the effectiveness of UEs is always positively related with the MI between clean and poisoned features.
• Figure 2: The drop of test accuracy (Acc Gap) and the reduction of MI (MI Gap) of various UEs on different models, including linear, 2-NN, 3-NN, LeNet-5(LN5), VGG-11(V11), ResNet-18(RN18). The results indicate that as the depth and complexity of the models increase, both the drop of test accuracy and the reduction of MI become more pronounced.
• Figure 3: Feature covariance of different unlearnable examples. Results show that all unlearnable methods reduce both MI and covariance, with our MI-UE achieving the lowest values.
• Figure 4: Visualization of MI-UE unlearnable noises and their corresponding clean and poisoned images on CIFAR-10. The first row is the clean images, the second row is the MI-UE noises, the last row is the poisoned images.
• Figure 5: Visualization of MI-UE unlearnable noises and their corresponding clean and poisoned images on ImageNet-subset. The first row is the clean images, the second row is the MI-UE noises, the last row is the poisoned images.

Theorems & Definitions (13)

• Theorem 5.1: Proof in Appendix \ref{['app:th-proof']}
• Remark 5.2
• Definition A.1: ${\mathcal{H}}$-Divergence, ben2010theory
• Definition A.2: ${\mathcal{H}}\Delta{\mathcal{H}}$ Space, ben2010theory
• Theorem A.3: Proof in Appendix \ref{['app:th-proof']}
• Remark A.4
• Remark A.5
• Remark A.6
• Lemma B.1: cover1999elements
• Theorem B.2: Restate of Theorem \ref{['th-gauss']}