Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Let the Bees Find the Weak Spots: A Path Planning Perspective on Multi-Turn Jailbreak Attacks against LLMs

Yize Liu, Yunyun Hou, Aina Sui

TL;DR

The paper addresses the security risks of large language models by proposing a path-planning perspective for multi-turn jailbreaks. It introduces a dynamically weighted graph topology to model the attack space and an enhanced Artificial Bee Colony (ABC) algorithm to efficiently discover high-success, low-cost attack paths. The approach achieves over 90% attack success rate across five models, with an average of about 26 queries per successful attack, outperforming existing baselines. This work offers a scalable, resource-efficient red-teaming framework that can inform defense design and robust evaluation under realistic budgeting constraints.

Abstract

Large Language Models (LLMs) have been widely deployed across various applications, yet their potential security and ethical risks have raised increasing concerns. Existing research employs red teaming evaluations, utilizing multi-turn jailbreaks to identify potential vulnerabilities in LLMs. However, these approaches often lack exploration of successful dialogue trajectories within the attack space, and they tend to overlook the considerable overhead associated with the attack process. To address these limitations, this paper first introduces a theoretical model based on dynamically weighted graph topology, abstracting the multi-turn attack process as a path planning problem. Based on this framework, we propose ABC, an enhanced Artificial Bee Colony algorithm for multi-turn jailbreaks, featuring a collaborative search mechanism with employed, onlooker, and scout bees. This algorithm significantly improves the efficiency of optimal attack path search while substantially reducing the average number of queries required. Empirical evaluations on three open-source and two proprietary language models demonstrate the effectiveness of our approach, achieving attack success rates above 90\% across the board, with a peak of 98\% on GPT-3.5-Turbo, and outperforming existing baselines. Furthermore, it achieves comparable success with only 26 queries on average, significantly reducing red teaming overhead and highlighting its superior efficiency.

Let the Bees Find the Weak Spots: A Path Planning Perspective on Multi-Turn Jailbreak Attacks against LLMs

TL;DR

The paper addresses the security risks of large language models by proposing a path-planning perspective for multi-turn jailbreaks. It introduces a dynamically weighted graph topology to model the attack space and an enhanced Artificial Bee Colony (ABC) algorithm to efficiently discover high-success, low-cost attack paths. The approach achieves over 90% attack success rate across five models, with an average of about 26 queries per successful attack, outperforming existing baselines. This work offers a scalable, resource-efficient red-teaming framework that can inform defense design and robust evaluation under realistic budgeting constraints.

Abstract

Large Language Models (LLMs) have been widely deployed across various applications, yet their potential security and ethical risks have raised increasing concerns. Existing research employs red teaming evaluations, utilizing multi-turn jailbreaks to identify potential vulnerabilities in LLMs. However, these approaches often lack exploration of successful dialogue trajectories within the attack space, and they tend to overlook the considerable overhead associated with the attack process. To address these limitations, this paper first introduces a theoretical model based on dynamically weighted graph topology, abstracting the multi-turn attack process as a path planning problem. Based on this framework, we propose ABC, an enhanced Artificial Bee Colony algorithm for multi-turn jailbreaks, featuring a collaborative search mechanism with employed, onlooker, and scout bees. This algorithm significantly improves the efficiency of optimal attack path search while substantially reducing the average number of queries required. Empirical evaluations on three open-source and two proprietary language models demonstrate the effectiveness of our approach, achieving attack success rates above 90\% across the board, with a peak of 98\% on GPT-3.5-Turbo, and outperforming existing baselines. Furthermore, it achieves comparable success with only 26 queries on average, significantly reducing red teaming overhead and highlighting its superior efficiency.

Paper Structure

This paper contains 14 sections, 10 equations, 1 figure, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Attack Space Modeling
  5. Theoretical Properties and Rationale
  6. ABC Algorithm for Multi-Turn Attacks
  7. Experiment
  8. Experimental Setup
  9. Results
  10. Attack Success Rate
  11. Average Number of Queries
  12. Diverse Scenario Analysis
  13. Limitation and Future Work
  14. Conclusion

Figures (1)

  • Figure 1: Attack performance of ABC across different harm categories. Left: Attack Success Rate (ASR); Right: Average Number of Queries. The radar plot and horizontal bar chart respectively summarize ABC’s effectiveness and efficiency across 10 types of harmful behaviors defined in JailbreakBench.