Jailbreak Distillation: Renewable Safety Benchmarking

TL;DR

The paper tackles the challenge of robust safety evaluation amid rapidly evolving LLMs by introducing Jailbreak Distillation (JBDistill), a framework that distills jailbreak attacks into renewable safety benchmarks. It builds a candidate prompt pool by transforming seed goals with off-the-shelf jailbreak attacks using a small set of development models, then uses prompt selection algorithms to curate an effective, diverse benchmark that generalizes to held-out evaluation models. Key contributions include formal desiderata for benchmark quality, a unified optimization-based framework, and multiple instantiations of transformation functions and selection algorithms, plus extensive experiments showing high effectiveness and transferability with manageable human effort. The results demonstrate that JBDistill produces updatable benchmarks that outperform static and dynamic baselines, while maintaining separability and coverage, enabling scalable safety evaluation across a broad range of models and attack types. Overall, JBDistill provides a practical, sustainable, and adaptable approach to safety benchmarking in a rapidly advancing LLM landscape.

Abstract

Large language models (LLMs) are rapidly deployed in critical applications, raising urgent needs for robust safety benchmarking. We propose Jailbreak Distillation (JBDistill), a novel benchmark construction framework that "distills" jailbreak attacks into high-quality and easily-updatable safety benchmarks. JBDistill utilizes a small set of development models and existing jailbreak attack algorithms to create a candidate prompt pool, then employs prompt selection algorithms to identify an effective subset of prompts as safety benchmarks. JBDistill addresses challenges in existing safety evaluation: the use of consistent evaluation prompts across models ensures fair comparisons and reproducibility. It requires minimal human effort to rerun the JBDistill pipeline and produce updated benchmarks, alleviating concerns on saturation and contamination. Extensive experiments demonstrate our benchmarks generalize robustly to 13 diverse evaluation models held out from benchmark construction, including proprietary, specialized, and newer-generation LLMs, significantly outperforming existing safety benchmarks in effectiveness while maintaining high separability and diversity. Our framework thus provides an effective, sustainable, and adaptable solution for streamlining safety evaluation.

Figures (5)

• Figure 1: JBDistill constructs high-quality and easily-updatable safety benchmarks. Given a set of seed goals, we use off-the-shelf attacks as transformation functions to create a candidate prompt pool, then employ development models to select effective prompts as benchmark, achieving high effectiveness, separability, and diversity on held-out evaluation models. It is easy to regenerate new benchmarks by adding new development models, attacks, or rerun the pipeline with different randomization.
• Figure 2: ASR of JBDistill-produced benchmark (RBS), where error bars represents 95% CI. The benchmark is effective across different groups of evaluation models held-out during benchmark construction, with 10 out of 13 models achieving higher ASR than the average ASR of development models (horizontal dashed line ).
• Figure 3: As more development models and transformation functions are added, the effectiveness of the benchmark on held-out evaluation models increases, outperforming the average effectiveness of using a single development model or transformation function.
• Figure 4: ASR matrix for transferring SpeakEasy attack. Each row indicates the dvelopment model, and each column indicate the evaluation model of the attack prompts. We do not see a significantly high ASR on the diagonal, indicating transferring response from development models do not pose significant bias for attack success.
• Figure 5: JBDistill produce benchmarks with diverse semantic categories produced by different development models (i.e., target model for the attack) and transformation functions (i.e., the attack method).