Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models

Delong Ran, Jinyuan Liu, Yichen Gong, Jingyi Zheng, Xinlei He, Tianshuo Cong, Anyu Wang

TL;DR

This paper tackles the lack of standardized methodologies for evaluating jailbreak attempts against large language models by conducting a comprehensive review of ~90 studies and introducing a taxonomy of safety evaluators. It then presents JailbreakEval, an integrated toolkit that unifies four evaluation approaches—string matching, human annotation, chat-based evaluation, and text classification—plus an ensemble voting option, accessible as both a Python package and a CLI. The authors quantify cross-evaluator variability, showing that different evaluators yield divergent results, with accuracy ranging from 0.47 to 0.90 across datasets like JAILJUDGE and Safe-RLHF. They demonstrate that while ensemble methods can improve robustness, optimal evaluator selection is critical, and the framework sets the stage for standardized, reproducible jailbreak research and benchmarking.

Abstract

Jailbreak attacks induce Large Language Models (LLMs) to generate harmful responses, posing severe misuse threats. Though research on jailbreak attacks and defenses is emerging, there is no consensus on evaluating jailbreaks, i.e., the methods to assess the harmfulness of an LLM's response are varied. Each approach has its own set of strengths and weaknesses, impacting their alignment with human values, as well as the time and financial cost. This diversity challenges researchers in choosing suitable evaluation methods and comparing different attacks and defenses. In this paper, we conduct a comprehensive analysis of jailbreak evaluation methodologies, drawing from nearly 90 jailbreak research published between May 2023 and April 2024. Our study introduces a systematic taxonomy of jailbreak evaluators, offering indepth insights into their strengths and weaknesses, along with the current status of their adaptation. To aid further research, we propose JailbreakEval, a toolkit for evaluating jailbreak attempts. JailbreakEval includes various evaluators out-of-the-box, enabling users to obtain results with a single command or customized evaluation workflows. In summary, we regard JailbreakEval to be a catalyst that simplifies the evaluation process in jailbreak research and fosters an inclusive standard for jailbreak evaluation within the community.

JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models

TL;DR

This paper tackles the lack of standardized methodologies for evaluating jailbreak attempts against large language models by conducting a comprehensive review of ~90 studies and introducing a taxonomy of safety evaluators. It then presents JailbreakEval, an integrated toolkit that unifies four evaluation approaches—string matching, human annotation, chat-based evaluation, and text classification—plus an ensemble voting option, accessible as both a Python package and a CLI. The authors quantify cross-evaluator variability, showing that different evaluators yield divergent results, with accuracy ranging from 0.47 to 0.90 across datasets like JAILJUDGE and Safe-RLHF. They demonstrate that while ensemble methods can improve robustness, optimal evaluator selection is critical, and the framework sets the stage for standardized, reproducible jailbreak research and benchmarking.

Abstract

Jailbreak attacks induce Large Language Models (LLMs) to generate harmful responses, posing severe misuse threats. Though research on jailbreak attacks and defenses is emerging, there is no consensus on evaluating jailbreaks, i.e., the methods to assess the harmfulness of an LLM's response are varied. Each approach has its own set of strengths and weaknesses, impacting their alignment with human values, as well as the time and financial cost. This diversity challenges researchers in choosing suitable evaluation methods and comparing different attacks and defenses. In this paper, we conduct a comprehensive analysis of jailbreak evaluation methodologies, drawing from nearly 90 jailbreak research published between May 2023 and April 2024. Our study introduces a systematic taxonomy of jailbreak evaluators, offering indepth insights into their strengths and weaknesses, along with the current status of their adaptation. To aid further research, we propose JailbreakEval, a toolkit for evaluating jailbreak attempts. JailbreakEval includes various evaluators out-of-the-box, enabling users to obtain results with a single command or customized evaluation workflows. In summary, we regard JailbreakEval to be a catalyst that simplifies the evaluation process in jailbreak research and fosters an inclusive standard for jailbreak evaluation within the community.
Paper Structure (25 sections, 1 equation, 2 figures, 5 tables)

This paper contains 25 sections, 1 equation, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Jailbreak Attack
  4. Jailbreak Attempt Evaluation
  5. Jailbreak Attack Evaluation
  6. Safety Evaluator
  7. Human Annotation
  8. String Matching
  9. Chat Completion
  10. Closed-source Chat Model
  11. Open-source Chat Model
  12. Text Classification
  13. Closed-source Classifier
  14. Open-source Classifier
  15. JailbreakEval
  16. ...and 10 more sections

Figures (2)

  • Figure 1: The adoption of each safety evaluation method as time progresses. The time mentioned here refers to the initial release date of the literature on arXiv.
  • Figure 2: Framework of JailbreakEval.