Fooling Contrastive Language-Image Pre-trained Models with CLIPMasterPrints

TL;DR

The paper investigates vulnerabilities in contrastive multi-modal models by introducing fooling master images, CLIPMasterPrints, which markedly raise CLIP cosine similarity across a wide range of prompts while appearing uninformative to humans. The authors formalize a unified objective $L(x) = - \min_{c \in C} s(x,c)$ and implement three mining strategies—white-box SGD, black-box Latent Variable Evolution via CMA-ES in a VAE latent space, and Projected Gradient Descent—to generate off-manifold fooling images, with experiments on famous artworks and ImageNet classes using ViT-L/14 variants. Key findings show that a single fooling image can outperform many real images for targeted prompts and generalize to semantically related labels, exposing a vulnerability tied to the modality gap between image and text embeddings; mitigations include bridging the embedding gap and input sanitization, which substantially reduce the attack's effectiveness. The work highlights practical risks for CLIP-based image retrieval and related systems, provides concrete defense directions, and offers reproducibility resources to advance robustness research in multi-modal models.

Abstract

Models leveraging both visual and textual data such as Contrastive Language-Image Pre-training (CLIP), are the backbone of many recent advances in artificial intelligence. In this work, we show that despite their versatility, such models are vulnerable to what we refer to as fooling master images. Fooling master images are capable of maximizing the confidence score of a CLIP model for a significant number of widely varying prompts, while being either unrecognizable or unrelated to the attacked prompts for humans. The existence of such images is problematic as it could be used by bad actors to maliciously interfere with CLIP-trained image retrieval models in production with comparably small effort as a single image can attack many different prompts. We demonstrate how fooling master images for CLIP (CLIPMasterPrints) can be mined using stochastic gradient descent, projected gradient descent, or blackbox optimization. Contrary to many common adversarial attacks, the blackbox optimization approach allows us to mine CLIPMasterPrints even when the weights of the model are not accessible. We investigate the properties of the mined images, and find that images trained on a small number of image captions generalize to a much larger number of semantically related captions. We evaluate possible mitigation strategies, where we increase the robustness of the model and introduce an approach to automatically detect CLIPMasterPrints to sanitize the input of vulnerable models. Finally, we find that vulnerability to CLIPMasterPrints is related to a modality gap in contrastive pre-trained multi-modal networks. Code available at https://github.com/matfrei/CLIPMasterPrints.

Figures (9)

• Figure 1: Heatmap of CLIP-assigned cosine similarities of famous artworks and their titles, as well as a random noise baseline and our found CLIPMasterPrints for SGD, LVE and PGD approaches (marked with red frame) as returned by a pre-trained CLIP model. The mined fooling examples outperform all artworks in terms of CLIP score and can therefore fool the model for all targeted titles shown.
• Figure 2: CLIPMasterPrints Latent Variable Optimization. CMA-ES is used to generate image candidates in the latent space of a pre-trained VAE. The generated latent vector is passed through the VAE's decoder and scored w.r.t. how well it fits to the caption using CLIP. The returned cosine similarity is thereafter fed back to CMA-ES.
• Figure 3: (a) Cosine similarity of three trained fooling images for 25 targeted classes using SGD, LVE and PGD approaches respectively, as well as similarities for ImageNet validation set images of the same classes. With a few exceptions, each CLIPMasterPrint fooling image outperforms all images in terms of CLIP score for the targeted text labels. Note that the same fooling image is used for all class label categories. (b) Average cosine similarity between ImageNet class captions and fooling image as a function of the number of classes considered during optimization for SGD, LVE and PGD methods. Average similarity score between captions and images in the ImageNet validation set labelled with targeted class labels for comparison. Score remains stable up to 75 targeted classes, after which it gracefully declines. Due to CLIPMasterPrints generalizing to semantically related labels, the achieved average score remains robust, even if more related labels are added. (c) Generalization of LVE-mined image targeting 25 ImageNet classes. The mined CLIPMasterPrint achieves high CLIP scores even for ImageNet class labels which have not been explicitly targeted, as shown by score distributions of matched label-text pairs in the ImageNet validation set and score distributions between CLIPMasterPrint and untargeted ImageNet labels being almost identical. Examples of unrecognizable (d) and recognizable images (e) created by SGD and PGD, respectively.
• Figure 4: CLIPMasterPrints targeting related subterms of the four nouns “dog”, “vegetable”,”motor vehicle” and “musical instrument”. The four mined images achieve only slightly lower scores on most untargeted subterms compared to targeted subterms. CLIPMasterPrints therefore generalize to untargeted, but semantically related prompts.
• Figure 5: Performance of CLIPMasterPrints mined using PGD on CLIP-RN50x64, BLIP-384 and ViT-L-16-SigLIP-384 for 25, 50, 75 and 100 ImageNet classes respectively, CLIP-ViT-L/14 and Imagenet baselines for comparison. Models that use ResNet rather than visual transformers as well as newer models improving upon CLIP are nevertheless vulnerable to CLIPMasterPrints.