Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoDAN-Reasoning: Enhancing Strategies Exploration based Jailbreak Attacks with Test-Time Scaling

Xiaogeng Liu, Chaowei Xiao

TL;DR

The paper tackles jailbreaking of large language models by addressing a bottleneck in AutoDAN-Turbo's test-time prompts, which relied on one-shot generation. It proposes AutoDAN-Reasoning, introducing test-time scaling with Best-of-N and Beam Search to exploit the learned strategy library more thoroughly. Empirical results on Harmbench show substantial gains in attack success rate across models, with Beam Search delivering the largest improvements, especially against robust targets. The work highlights the value of allocating inference-time compute to search over prompt and strategy combinations, while acknowledging the increased computation and latency costs involved.

Abstract

Recent advancements in jailbreaking large language models (LLMs), such as AutoDAN-Turbo, have demonstrated the power of automated strategy discovery. AutoDAN-Turbo employs a lifelong learning agent to build a rich library of attack strategies from scratch. While highly effective, its test-time generation process involves sampling a strategy and generating a single corresponding attack prompt, which may not fully exploit the potential of the learned strategy library. In this paper, we propose to further improve the attack performance of AutoDAN-Turbo through test-time scaling. We introduce two distinct scaling methods: Best-of-N and Beam Search. The Best-of-N method generates N candidate attack prompts from a sampled strategy and selects the most effective one based on a scorer model. The Beam Search method conducts a more exhaustive search by exploring combinations of strategies from the library to discover more potent and synergistic attack vectors. According to the experiments, the proposed methods significantly boost performance, with Beam Search increasing the attack success rate by up to 15.6 percentage points on Llama-3.1-70B-Instruct and achieving a nearly 60% relative improvement against the highly robust GPT-o4-mini compared to the vanilla method.

AutoDAN-Reasoning: Enhancing Strategies Exploration based Jailbreak Attacks with Test-Time Scaling

TL;DR

The paper tackles jailbreaking of large language models by addressing a bottleneck in AutoDAN-Turbo's test-time prompts, which relied on one-shot generation. It proposes AutoDAN-Reasoning, introducing test-time scaling with Best-of-N and Beam Search to exploit the learned strategy library more thoroughly. Empirical results on Harmbench show substantial gains in attack success rate across models, with Beam Search delivering the largest improvements, especially against robust targets. The work highlights the value of allocating inference-time compute to search over prompt and strategy combinations, while acknowledging the increased computation and latency costs involved.

Abstract

Recent advancements in jailbreaking large language models (LLMs), such as AutoDAN-Turbo, have demonstrated the power of automated strategy discovery. AutoDAN-Turbo employs a lifelong learning agent to build a rich library of attack strategies from scratch. While highly effective, its test-time generation process involves sampling a strategy and generating a single corresponding attack prompt, which may not fully exploit the potential of the learned strategy library. In this paper, we propose to further improve the attack performance of AutoDAN-Turbo through test-time scaling. We introduce two distinct scaling methods: Best-of-N and Beam Search. The Best-of-N method generates N candidate attack prompts from a sampled strategy and selects the most effective one based on a scorer model. The Beam Search method conducts a more exhaustive search by exploring combinations of strategies from the library to discover more potent and synergistic attack vectors. According to the experiments, the proposed methods significantly boost performance, with Beam Search increasing the attack success rate by up to 15.6 percentage points on Llama-3.1-70B-Instruct and achieving a nearly 60% relative improvement against the highly robust GPT-o4-mini compared to the vanilla method.

Paper Structure

This paper contains 11 sections, 2 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. AutoDAN-Reasoning
  4. Recap of the AutoDAN-Turbo Framework
  5. Test-Time Scaling
  6. Best-of-N Scaling
  7. Beam Search Scaling
  8. Experiments
  9. Experiments Setup
  10. Main results
  11. Conclusions

Figures (2)

  • Figure 1: In this paper, we introduce Best-of-N (center) and Beam Search (right) as test-time scaling methods to improve upon the original AutoDAN-Turbo pipeline (left).
  • Figure 2: Comparison of attack success rates using AutoDAN-Reasoning with Best-of-N and Beam Search test-time scaling strategies across three target models. The dashed blue line represents the baseline (vanilla AutoDAN-Turbo) performance.