Zero-Shot Class Unlearning in CLIP with Synthetic Samples

TL;DR

This paper tackles the challenge of unlearning in CLIP without access to training data by introducing a zero-shot forgetting framework that generates synthetic forget samples via gradient ascent and applies local Lipschitz regularization to both image and text encoders. The approach extends Lipschitz-based forgetting from unimodal vision models to the multimodal CLIP architecture, using the perturbed image as a proxy for text representations and updating only a targeted subset of layers in an iterative fashion until forgetting meets a defined threshold. Through extensive experiments on four fine-grained datasets, the authors demonstrate that their Lip method effectively reduces target-class accuracy while preserving performance on non-forget classes and across other datasets, with thorough ablations validating the necessity of joint visual-textual losses and synthetic data generation. The work advances practical privacy-preserving capabilities for large multimodal models and offers a scalable, data-free forgetting mechanism suitable for dynamic deployment scenarios.

Abstract

Machine unlearning is a crucial area of research. It is driven by the need to remove sensitive information from models to safeguard individuals' right to be forgotten under rigorous regulations such as GDPR. In this work, we focus on unlearning within CLIP, a dual vision-language encoder model trained on a massive dataset of image-text pairs using contrastive loss. To achieve forgetting we expand the application of Lipschitz regularization to the multimodal context of CLIP. Specifically, we ensure the smoothing of both visual and textual embeddings associated with the class intended to be forgotten relative to the perturbation introduced to the samples from that class. Additionally, importantly, we remove the necessity for real forgetting data by generating synthetic samples through gradient ascent maximizing the target class. Our forgetting procedure is iterative, where we track accuracy on a synthetic forget set and stop when accuracy falls below a chosen threshold. We employ a selective layers update strategy based on their average absolute gradient value to mitigate over-forgetting. We validate our approach on several standard datasets and provide thorough ablation analysis and comparisons with previous work.

Figures (5)

• Figure 1: Overview of the approach. First, we generate synthetic images of a class to forget by gradient ascent. Then, we perform a Gaussian perturbation of the images and pass the original and perturbed images through CLIP image encoder and textual description of the class through CLIP textual encoder. As image and text are projected into a shared embedding space, we can use the final representation of the perturbed image as a perturbed representation of text "Picture of a dog". Finally, we apply Lipschitz regularization and backpropagate to selected layers based on their importance to visual and textual encoders.
• Figure 2: Comparing the average scores of unlearning with Lip method using Synthetic vs Real data.
• Figure 3: Predictions of the model before (BF) and after forgetting (AF) with the prediction BF representing the target class to forget.
• Figure 4: Synthetic images examples.
• Figure 5: Selected layers for forgetting. The figure shows the top 25 most frequent updated layers during forgetting process across selected classes and datasets.