Fast Certification of Vision-Language Models Using Incremental Randomized Smoothing

TL;DR

This work tackles the lack of robustness guarantees for zero-shot vision-language models with open vocabulary prompts by introducing Open Vocabulary Certification (OVC), a framework that leverages incremental randomized smoothing and pre-computed certificates to certify novel prompts efficiently. It proposes three methods—Modified-IRS, Cached-OVC, and MVN-OVC—to accelerate certification for new prompts, with MVN-OVC offering substantial speedups through a Gaussian-approximation of the embedding space. Empirical evaluations on CLIP and OpenCLIP across ImageNet and CIFAR-10 show that the certificates match standard RS radii while dramatically reducing computation and memory requirements. The approach enables practical, scalable robustness guarantees for open-vocabulary vision-language systems, facilitating safer deployment in the wild.

Abstract

A key benefit of deep vision-language models such as CLIP is that they enable zero-shot open vocabulary classification; the user has the ability to define novel class labels via natural language prompts at inference time. However, while CLIP-based zero-shot classifiers have demonstrated competitive performance across a range of domain shifts, they remain highly vulnerable to adversarial attacks. Therefore, ensuring the robustness of such models is crucial for their reliable deployment in the wild. In this work, we introduce Open Vocabulary Certification (OVC), a fast certification method designed for open-vocabulary models like CLIP via randomized smoothing techniques. Given a base "training" set of prompts and their corresponding certified CLIP classifiers, OVC relies on the observation that a classifier with a novel prompt can be viewed as a perturbed version of nearby classifiers in the base training set. Therefore, OVC can rapidly certify the novel classifier using a variation of incremental randomized smoothing. By using a caching trick, we achieve approximately two orders of magnitude acceleration in the certification process for novel prompts. To achieve further (heuristic) speedups, OVC approximates the embedding space at a given input using a multivariate normal distribution bypassing the need for sampling via forward passes through the vision backbone. We demonstrate the effectiveness of OVC on through experimental evaluation using multiple vision-language backbones on the CIFAR-10 and ImageNet test datasets.

Figures (18)

• Figure 1: Workflow of OVC. For a given prompt, using relatively few samples, we find a prompt (out of prompts whose prediction is known) which is most similar in prediction to the given prompt. If the difference in prediction is below certain threshold, we certify using the information from existing prompt, saving time.
• Figure 2: Scatter plot showing the relationship between probability of top-most class, $p_A$, for prompts with varying degree of similarity. Even when the two prompts are very close in cosine similarity, Fig. (a), they vary widely on the probability for the top-most class, indicating that IRS ugare2023incremental can not be applied directly for OVC. All the certificates have been computed for CLIP-RN50 on ImageNet with $\sigma = 0.25$
• Figure 3: Plot shows agreement in predictions for the input, for a novel prompt with predictions made by existing prompts. For $10K$ random perturbations with $\sigma=0.25$, among $500$ input samples tested, we found that for about $30\%$, there is at least one prompt for which disagreement is $< 1\%$.
• Figure 4: Certification of CLIP-RN50 model for various $\sigma$ for ImageNet and CIFAR-10 dataset.
• Figure 5: Scatter plot comparing radius obtained using Modified-IRS and standard method (Cohen et al), for ImageNet & CIFAR-10 for CLIP-RN50 with $\sigma=0.25$.

Theorems & Definitions (4)

• Theorem 1
• Theorem 2
• Remark 1
• Remark 2