LLM Safeguard is a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attacks
Qingzhao Zhang, Ziyang Xiong, Z. Morley Mao
TL;DR
This paper reveals a previously underappreciated DoS risk in LLM safeguards: false positives can cause safe user prompts to be rejected, degrading service and potentially harming users. It formulates four attack models—white-box and black-box prompt injection, poisoned fine-tuning, and poisoned fine-tuning with backdoors—and evaluates them across diverse datasets and models, including Llama Guard 3 and GPT-family systems. Key findings show high attack success under realistic conditions (e.g., up to 97% denial with short prompts on Llama Guard 3) and notable differences in resilience among models and attack types, underscoring the need for evaluating safeguarding systems against false positives. The work highlights practical implications for safeguarding design, suggests mitigation trade-offs, and provides a foundation for incorporating false-positive robustness into standard LLM safety assessments.
Abstract
Safety is a paramount concern for large language models (LLMs) in open deployment, motivating the development of safeguard methods that enforce ethical and responsible use through safety alignment or guardrail mechanisms. Jailbreak attacks that exploit the \emph{false negatives} of safeguard methods have emerged as a prominent research focus in the field of LLM security. However, we found that the malicious attackers could also exploit false positives of safeguards, i.e., fooling the safeguard model to block safe content mistakenly, leading to a denial-of-service (DoS) affecting LLM users. To bridge the knowledge gap of this overlooked threat, we explore multiple attack methods that include inserting a short adversarial prompt into user prompt templates and corrupting the LLM on the server by poisoned fine-tuning. In both ways, the attack triggers safeguard rejections of user requests from the client. Our evaluation demonstrates the severity of this threat across multiple scenarios. For instance, in the scenario of white-box adversarial prompt injection, the attacker can use our optimization process to automatically generate seemingly safe adversarial prompts, approximately only 30 characters long, that universally block over 97% of user requests on Llama Guard 3. These findings reveal a new dimension in LLM safeguard evaluation -- adversarial robustness to false positives.