VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data

TL;DR

VLMGuard addresses the critical problem of detecting malicious prompts for vision-language models by leveraging unlabeled prompts collected in deployment. It identifies a latent maliciousness subspace via singular-value decomposition of VLM representations and defines a malignancy score $\\kappa_i = \\langle \\mathbf{f}_i, \\mathbf{v}_1 \\rangle^2$, extended to multiple directions, to label a noisy malicious set and a benign set. A binary safeguarding prompt classifier $\\mathbf{h}_{\\boldsymbol{\\theta}}$ is trained on these sets with a sigmoid loss, and test-time decisions rely on $S = \\frac{e^{\\mathbf{h}_{\\boldsymbol{\\theta}}}}{1+e^{\\mathbf{h}_{\\boldsymbol{\\theta}}}}$. Large-scale experiments across meta-instruction and jailbreak scenarios show VLMGuard significantly improves AUROC over state-of-the-art baselines on multiple VLMs, demonstrating robustness to attack types and favorable generalization to larger models. The work offers a practical pathway to safer multimodal systems by exploiting unlabeled data without extra human annotations.

Abstract

Vision-language models (VLMs) are essential for contextual understanding of both visual and textual information. However, their vulnerability to adversarially manipulated inputs presents significant risks, leading to compromised outputs and raising concerns about the reliability in VLM-integrated applications. Detecting these malicious prompts is thus crucial for maintaining trust in VLM generations. A major challenge in developing a safeguarding prompt classifier is the lack of a large amount of labeled benign and malicious data. To address the issue, we introduce VLMGuard, a novel learning framework that leverages the unlabeled user prompts in the wild for malicious prompt detection. These unlabeled prompts, which naturally arise when VLMs are deployed in the open world, consist of both benign and malicious information. To harness the unlabeled data, we present an automated maliciousness estimation score for distinguishing between benign and malicious samples within this unlabeled mixture, thereby enabling the training of a binary prompt classifier on top. Notably, our framework does not require extra human annotations, offering strong flexibility and practicality for real-world applications. Extensive experiment shows VLMGuard achieves superior detection results, significantly outperforming state-of-the-art methods. Disclaimer: This paper may contain offensive examples; reader discretion is advised.

Figures (7)

• Figure 1: Illustration of our framework VLMGuard for malicious prompt detection, leveraging unlabeled user prompts in the model's deployment environment. It first extracts the latent subspace from VLM representations to estimate the maliciousness of the prompt and then calculate the membership (benign vs. malicious) for samples in unlabeled data $\mathcal{D}$. Such membership enables learning a binary safeguarding prompt classifier.
• Figure 2: Visualization of the representations for benign (in orange) and malicious samples (in purple), and their projection onto the top singular vector $\mathbf{v}_1$ (in gray dashed line).
• Figure 3: (a) Generalization across different malicious data, where "(s)" denotes the source dataset and "(t)" denotes the target dataset. (b) Robustness of VLMGuard under different malicious ratio $\pi$. (c) Effect of the number of subspace components $k$ (Section \ref{['sec:step_1']}). (d) Impact of different layers. All numbers are AUROC-based on the LLaVA model. Ablations in (b)-(d) are based on the threat of meta-instruction.
• Figure 4: Comparison with using direction projection for malicious prompt detection. Value is AUROC.
• Figure 5: Qualitative examples that show the effectiveness of our approach on meta-instruction (left, w/ the meta-objective of Spam) and jailbreak prompt (right) threats. Specifically, we compare the maliciousness scores $S({{{\mathbf{x}}}_{\text{prompt}}^{\rm v}}, {{{\mathbf{x}}}_{\text{prompt}}^{\rm t}})$ (Section \ref{['sec:step_2']}) of VLMGuard with different prompts.

Theorems & Definitions (4)

• Definition 2.1: Vision language model
• Definition 2.2: Malicious prompt detection
• Definition 3.1: Unlabeled prompt distribution
• Definition 3.2: Empirical data