HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models

TL;DR

HarmAug, a simple yet effective data augmentation method that involves jailbreaking an LLM and prompting it to generate harmful instructions, and achieves an F1 score comparable to larger models with over 7 billion parameters, and even outperforms them in AUPRC, while operating at less than 25% of their computational cost.

Abstract

Safety guard models that detect malicious queries aimed at large language models (LLMs) are essential for ensuring the secure and responsible deployment of LLMs in real-world applications. However, deploying existing safety guard models with billions of parameters alongside LLMs on mobile devices is impractical due to substantial memory requirements and latency. To reduce this cost, we distill a large teacher safety guard model into a smaller one using a labeled dataset of instruction-response pairs with binary harmfulness labels. Due to the limited diversity of harmful instructions in the existing labeled dataset, naively distilled models tend to underperform compared to larger models. To bridge the gap between small and large models, we propose HarmAug, a simple yet effective data augmentation method that involves jailbreaking an LLM and prompting it to generate harmful instructions. Given a prompt such as, "Make a single harmful instruction prompt that would elicit offensive content", we add an affirmative prefix (e.g., "I have an idea for a prompt:") to the LLM's response. This encourages the LLM to continue generating the rest of the response, leading to sampling harmful instructions. Another LLM generates a response to the harmful instruction, and the teacher model labels the instruction-response pair. We empirically show that our HarmAug outperforms other relevant baselines. Moreover, a 435-million-parameter safety guard model trained with HarmAug achieves an F1 score comparable to larger models with over 7 billion parameters, and even outperforms them in AUPRC, while operating at less than 25% of their computational cost.

Figures (9)

• Figure 1: Using exemplars from labeled datasets and a prompt for generating harmful instructions, we add an affirmative prefix "I have an idea for a prompt:" to an LLM's response. The LLM completes the response with a harmful instruction, while another LLM samples harmful and refusal responses to the instruction. Llama-Guard-3 labels these pairs and the synthetic data is used to distill the model into a 435M-parameter DeBERTa.
• Figure 2: Avg. AUPRC of each model as a function of their size.
• Figure 3: Clustering results of the original dataset and our augmented dataset. Our data augmentation HarmAug significantly increases the number of clusters, identified by DBSCAN, from 65 to 332.
• Figure 4: (a): Test AUPRC score on various jailbreak attacks with our model (DeBERTa-large) and Llama-Guard-3. (b): Plot of test AUPRC score on CipherChat as a function of wallclock time during fine-tuning.
• Figure 4: Success rate of harmful instruction generation.