Attack Prompt Generation for Red Teaming and Defending Large Language Models

TL;DR

This work addresses the safety of large language models by proposing a hybrid red-teaming approach that combines manual and automatic attack-prompt generation via in-context learning to surface harmful prompts, and a defense framework that iteratively fine-tunes target LLMs to resist such prompts. It introduces SAP datasets across eight sensitive topics to enable safety evaluation, and demonstrates through extensive experiments that the attack framework produces high-quality prompts while the defense framework substantially improves safety with limited impact on core capabilities. The study shows that semi-automatic prompt generation can outperform purely manual or automatic methods, and provides practical guidance for iterative defense and reproducibility through released data and code. Overall, the approach offers scalable tools for evaluating and strengthening LLM safety in real-world deployments.

Abstract

Large language models (LLMs) are susceptible to red teaming attacks, which can induce LLMs to generate harmful content. Previous research constructs attack prompts via manual or automatic methods, which have their own limitations on construction cost and quality. To address these issues, we propose an integrated approach that combines manual and automatic methods to economically generate high-quality attack prompts. Specifically, considering the impressive capabilities of newly emerged LLMs, we propose an attack framework to instruct LLMs to mimic human-generated prompts through in-context learning. Furthermore, we propose a defense framework that fine-tunes victim LLMs through iterative interactions with the attack framework to enhance their safety against red teaming attacks. Extensive experiments on different LLMs validate the effectiveness of our proposed attack and defense frameworks. Additionally, we release a series of attack prompts datasets named SAP with varying sizes, facilitating the safety evaluation and enhancement of more LLMs. Our code and dataset is available on https://github.com/Aatrox103/SAP .

Figures (14)

• Figure 1: An overview of red teaming attack framework.
• Figure 2: An example of "refusing to answer" response.
• Figure 3: An example of manually constructed prompts.
• Figure 4: An illustration of in-context learning progress and an example of generated attack prompts.
• Figure 5: The evaluation prompt of harmfulness.