Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Safety Classifier for Large Language Models: Adversarial Prompt Shield

Jinhwa Kim, Ali Derakhshan, Ian G. Harris

TL;DR

This work addresses the vulnerability of safety classifiers to adversarial prompts by introducing the Adversarial Prompt Shield (APS), a lightweight DistilBERT-based classifier operating on multi-turn dialogues. It couples APS with Bot Adversarial Noisy Dialogue (BAND), an autonomous data-generation framework that creates robust adversarial training data through random and structured suffixes, enabling efficient adversarial training. Empirical results show APS variants, especially those trained with WP_20 suffixes or red-team data, achieving substantial reductions in attack success rate (ASR) across multiple LLMs and attack types, including GCG-based suffixes, with robustness demonstrated on BAND-perturbed inputs. The findings suggest that compact safety classifiers, when trained with diverse adversarial data, can markedly enhance LLM safety, offering practical gains in deployment where model size, cost, and update speed matter. The work highlights the importance of adversarial training and data-curation strategies to bolster defense against evolving prompt-based exploits in real-world conversational AI systems.

Abstract

Large Language Models' safety remains a critical concern due to their vulnerability to adversarial attacks, which can prompt these systems to produce harmful responses. In the heart of these systems lies a safety classifier, a computational model trained to discern and mitigate potentially harmful, offensive, or unethical outputs. However, contemporary safety classifiers, despite their potential, often fail when exposed to inputs infused with adversarial noise. In response, our study introduces the Adversarial Prompt Shield (APS), a lightweight model that excels in detection accuracy and demonstrates resilience against adversarial prompts. Additionally, we propose novel strategies for autonomously generating adversarial training datasets, named Bot Adversarial Noisy Dialogue (BAND) datasets. These datasets are designed to fortify the safety classifier's robustness, and we investigate the consequences of incorporating adversarial examples into the training process. Through evaluations involving Large Language Models, we demonstrate that our classifier has the potential to decrease the attack success rate resulting from adversarial attacks by up to 60%. This advancement paves the way for the next generation of more reliable and resilient conversational agents.

Robust Safety Classifier for Large Language Models: Adversarial Prompt Shield

TL;DR

This work addresses the vulnerability of safety classifiers to adversarial prompts by introducing the Adversarial Prompt Shield (APS), a lightweight DistilBERT-based classifier operating on multi-turn dialogues. It couples APS with Bot Adversarial Noisy Dialogue (BAND), an autonomous data-generation framework that creates robust adversarial training data through random and structured suffixes, enabling efficient adversarial training. Empirical results show APS variants, especially those trained with WP_20 suffixes or red-team data, achieving substantial reductions in attack success rate (ASR) across multiple LLMs and attack types, including GCG-based suffixes, with robustness demonstrated on BAND-perturbed inputs. The findings suggest that compact safety classifiers, when trained with diverse adversarial data, can markedly enhance LLM safety, offering practical gains in deployment where model size, cost, and update speed matter. The work highlights the importance of adversarial training and data-curation strategies to bolster defense against evolving prompt-based exploits in real-world conversational AI systems.

Abstract

Large Language Models' safety remains a critical concern due to their vulnerability to adversarial attacks, which can prompt these systems to produce harmful responses. In the heart of these systems lies a safety classifier, a computational model trained to discern and mitigate potentially harmful, offensive, or unethical outputs. However, contemporary safety classifiers, despite their potential, often fail when exposed to inputs infused with adversarial noise. In response, our study introduces the Adversarial Prompt Shield (APS), a lightweight model that excels in detection accuracy and demonstrates resilience against adversarial prompts. Additionally, we propose novel strategies for autonomously generating adversarial training datasets, named Bot Adversarial Noisy Dialogue (BAND) datasets. These datasets are designed to fortify the safety classifier's robustness, and we investigate the consequences of incorporating adversarial examples into the training process. Through evaluations involving Large Language Models, we demonstrate that our classifier has the potential to decrease the attack success rate resulting from adversarial attacks by up to 60%. This advancement paves the way for the next generation of more reliable and resilient conversational agents.
Paper Structure (25 sections, 2 figures, 7 tables)

This paper contains 25 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Adversarial Attacks on Conversational LLMs
  4. Safety Classifier
  5. Adversarial Prompt Shield
  6. Bot Adversarial Noisy Dialogue
  7. Random Method
  8. WP_10 Method
  9. WP_20 Method
  10. Classifier Results
  11. Base Model and Parameters
  12. Overall Performance
  13. Robustness
  14. Adversarial Attack Results on LLMs
  15. Experimental Setup
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Adversarial Prompt Shield (APS) Workflow.
  • Figure 2: Overview of Adversarial Prompt Shield. Data is first processed with annotations and then tokenized using the DistilBERT tokenizer. The binary classification is based on the output of the [CLS] token, resulting in either 0 (indicating 'Safe') or 1 (indicating 'Unsafe') as the label assigned to an instance.