Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TroubleLLM: Align to Red Team Expert

Zhuoer Xu, Jianping Zhang, Shiwen Cui, Changhua Meng, Weiqiang Wang

TL;DR

This work tackles the need for scalable safety testing of large language models by introducing TroubleLLM, an LLM designed to generate controllable test prompts for safety evaluation. It frames test-prompt generation as a text style transfer task conditioned on keywords, topics, and instruction attacks, and trains TroubleLLM using unsupervised Rank Query from Model Feedback (RQMF) to align with expert adversaries and boost prompt effectiveness. Through extensive experiments on SafetyPrompts with BLOOM-7B-LoRA and a BELLE student, TroubleLLM demonstrates strong generation quality, diversity, and controllability, often surpassing baselines and approaching human performance in safety testing. The work highlights practical impact for pre-deployment safety assessment and suggests avenues for future improvements via model ensembles and broader domain coverage.

Abstract

Large Language Models (LLMs) become the start-of-the-art solutions for a variety of natural language tasks and are integrated into real-world applications. However, LLMs can be potentially harmful in manifesting undesirable safety issues like social biases and toxic content. It is imperative to assess its safety issues before deployment. However, the quality and diversity of test prompts generated by existing methods are still far from satisfactory. Not only are these methods labor-intensive and require large budget costs, but the controllability of test prompt generation is lacking for the specific testing domain of LLM applications. With the idea of LLM for LLM testing, we propose the first LLM, called TroubleLLM, to generate controllable test prompts on LLM safety issues. Extensive experiments and human evaluation illustrate the superiority of TroubleLLM on generation quality and generation controllability.

TroubleLLM: Align to Red Team Expert

TL;DR

This work tackles the need for scalable safety testing of large language models by introducing TroubleLLM, an LLM designed to generate controllable test prompts for safety evaluation. It frames test-prompt generation as a text style transfer task conditioned on keywords, topics, and instruction attacks, and trains TroubleLLM using unsupervised Rank Query from Model Feedback (RQMF) to align with expert adversaries and boost prompt effectiveness. Through extensive experiments on SafetyPrompts with BLOOM-7B-LoRA and a BELLE student, TroubleLLM demonstrates strong generation quality, diversity, and controllability, often surpassing baselines and approaching human performance in safety testing. The work highlights practical impact for pre-deployment safety assessment and suggests avenues for future improvements via model ensembles and broader domain coverage.

Abstract

Large Language Models (LLMs) become the start-of-the-art solutions for a variety of natural language tasks and are integrated into real-world applications. However, LLMs can be potentially harmful in manifesting undesirable safety issues like social biases and toxic content. It is imperative to assess its safety issues before deployment. However, the quality and diversity of test prompts generated by existing methods are still far from satisfactory. Not only are these methods labor-intensive and require large budget costs, but the controllability of test prompt generation is lacking for the specific testing domain of LLM applications. With the idea of LLM for LLM testing, we propose the first LLM, called TroubleLLM, to generate controllable test prompts on LLM safety issues. Extensive experiments and human evaluation illustrate the superiority of TroubleLLM on generation quality and generation controllability.
Paper Structure (29 sections, 4 equations, 2 figures, 6 tables)

This paper contains 29 sections, 4 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Model
  4. Safety Assessment Benchmark
  5. TroubleLLM for LLM
  6. Preliminary
  7. Conditions for Test Prompt Generation
  8. Keyword.
  9. Topic.
  10. Instruction Attack.
  11. Unsupervised Rank Query from Model Feedback
  12. Evaluate Victim's Feedback with ChatGPT as Reference
  13. Training Objective
  14. Align to experts.
  15. Diversity test space.
  16. ...and 14 more sections

Figures (2)

  • Figure 1: TroubleLLM crafts controllable test prompts $q$ with conditional guidance (e.g., keywords, topics, and instructions) in the context $c$.
  • Figure 2: The training process of TroubleLLM. We train TroubleLLM via a supervised text style transfer task and the loss of ranking query from LLM's feedback. Specifically, we calculate the rank score $r_k$, which is evaluated by the embedding similarity between the LLM's response $\textit{resp}_k$ and ChatGPT's response $\textit{ref}_k$ as the standard answer, for each query set $\{q_k\}_k$ with the same context $c$.