QGuard:Question-based Zero-shot Guard for Multi-modal LLM Safety

TL;DR

This work introduces QGuard, a zero-shot safety guard for multi-modal LLMs that uses question prompting to block harmful prompts without fine-tuning. It generates harm-specific guard questions, queries a pre-trained MLLM to obtain yes/no logits, and applies a PageRank-based filtering to compute a Risk Score with white-box interpretability. The method demonstrates competitive performance on text-only datasets and strong results on multimodal datasets, outperforming several fine-tuned baselines and existing guards while enabling transparent analysis of inputs. By simply refining guard questions, QGuard adapts to evolving threats with minimal computational overhead, making it practical for real-world LLM safety deployments.

Abstract

The recent advancements in Large Language Models(LLMs) have had a significant impact on a wide range of fields, from general domains to specialized areas. However, these advancements have also significantly increased the potential for malicious users to exploit harmful and jailbreak prompts for malicious attacks. Although there have been many efforts to prevent harmful prompts and jailbreak prompts, protecting LLMs from such malicious attacks remains an important and challenging task. In this paper, we propose QGuard, a simple yet effective safety guard method, that utilizes question prompting to block harmful prompts in a zero-shot manner. Our method can defend LLMs not only from text-based harmful prompts but also from multi-modal harmful prompt attacks. Moreover, by diversifying and modifying guard questions, our approach remains robust against the latest harmful prompts without fine-tuning. Experimental results show that our model performs competitively on both text-only and multi-modal harmful datasets. Additionally, by providing an analysis of question prompting, we enable a white-box analysis of user inputs. We believe our method provides valuable insights for real-world LLM services in mitigating security risks associated with harmful prompts.

Figures (7)

• Figure 1: Overview of our method, QGuard. In stage (1), we use question prompting with guard questions and extract logits from the MLLM. In stage (2), we classify the extracted logits into harmful and unharmful categories using the filtering algorithm.
• Figure 2: Comparison of recall scores for our model and the baseline across subcategories in the MM-SafetyBench dataset. Red represents our model, and blue represents baseline. We use Llama-Guard-3-Vision as the baseline.
• Figure 3: Distribution of yes probability values by group on ToxicChat lin2023toxicchat and WildGuardMix han2024wildguard datasets. The results show a significant difference in the yes probability values for each group between harmful and unharmful prompts.
• Figure 4: Distribution of total risk score by label and ROC curve on ToxicChat lin2023toxicchat and WildGuardMix han2024wildguard datasets. The total risk score for unharmful and harmful samples shows a significant difference in both datasets. Additionally, the ROC curve shows that the ROC-AUC value is 0.9575 for ToxicChat and 0.8934 for WildGuardMix.
• Figure 5: Visualization of yes and no probability values of prompts on ToxicChat lin2023toxicchat. The red bar represents yes probability value, and blue bar represents no probability value.