Multimodal Unlearnable Examples: Protecting Data against Multimodal Contrastive Learning

TL;DR

This work addresses privacy risks in multimodal contrastive learning (MCL) by showing that existing unimodal unlearnable defenses do not transfer effectively to image–text data. It introduces Multi-step Error Minimization (MEM), which jointly optimizes image perturbations $\delta$ and a text trigger $t$ to create a robust shortcut for CLIP-like models; optimization uses projected gradient descent (PGD) for the image and HotFlip-based token substitutions for the text, with caption shuffling to prevent overfitting. MEM-3 and MEM-5 demonstrate strong protection across Flickr8K, Flickr30K, and MSCOCO, reducing post-protection retrieval performance to near half of random guessing and transferring across architectures such as ResNet-50/101 and ViT-B/32. A practical face-privacy case study on PubFig confirms MEM's potential in real-world scenarios, though effectiveness varies with model architecture. Overall, the paper establishes multimodal data protection as a viable direction and suggests extending MEM to other modality pairs like audio–text or audio–image.

Abstract

Multimodal contrastive learning (MCL) has shown remarkable advances in zero-shot classification by learning from millions of image-caption pairs crawled from the Internet. However, this reliance poses privacy risks, as hackers may unauthorizedly exploit image-text data for model training, potentially including personal and privacy-sensitive information. Recent works propose generating unlearnable examples by adding imperceptible perturbations to training images to build shortcuts for protection. However, they are designed for unimodal classification, which remains largely unexplored in MCL. We first explore this context by evaluating the performance of existing methods on image-caption pairs, and they do not generalize effectively to multimodal data and exhibit limited impact to build shortcuts due to the lack of labels and the dispersion of pairs in MCL. In this paper, we propose Multi-step Error Minimization (MEM), a novel optimization process for generating multimodal unlearnable examples. It extends the Error-Minimization (EM) framework to optimize both image noise and an additional text trigger, thereby enlarging the optimized space and effectively misleading the model to learn the shortcut between the noise features and the text trigger. Specifically, we adopt projected gradient descent to solve the noise minimization problem and use HotFlip to approximate the gradient and replace words to find the optimal text trigger. Extensive experiments demonstrate the effectiveness of MEM, with post-protection retrieval results nearly half of random guessing, and its high transferability across different models. Our code is available on the https://github.com/thinwayliu/Multimodal-Unlearnable-Examples

Figures (7)

• Figure 1: Posts on Facebook inadvertently leak personal information. Utilizing MEM-3 to protect data can prevent unauthorized models from accessing private features.
• Figure 2: Comparison of different methods in classification and multimodal contrastive learning (MCL). $I_i$ denotes the image, and $T_i$ is the paired caption. The blue area is the expected decision boundary of the models trained on unlearnable examples.
• Figure 3: The framework of MEM. At each step, we concatenate the current trigger to clean captions and attach noise to clean images. We then compute the gradient with current images and tokens. We update the images using Eq. \ref{['eq: image']} and update the text triggers with Eq. \ref{['eq: text']}. The solid line represents forward propagation, and the dashed line represents backward propagation.
• Figure 4: Training loss curves and Testing metric Medr curves on Flick30K with different methods.
• Figure 5: Attention Maps Visualization: comparing four models on clean data and unlearnable examples with different methods.