An Automated Framework for Strategy Discovery, Retrieval, and Evolution in LLM Jailbreak Attacks

TL;DR

The paper addresses the security risks posed by jailbreak attacks against LLMs by introducing ASTRA, a modular automated framework that continuously discovers, evaluates, distills, and reuses attack strategies in a closed loop. It couples an Attack Designer, a Strategy Extractor, and a Strategy Storage & Retrieval system to build three evolving strategy libraries (Effective, Promising, Ineffective) and uses vector-based retrieval to guide prompts without manual templates. Across nine target LLMs and HarmBench/AdvBench-50 datasets, ASTRA achieves a mean ASR of 82.7% and an average of 2.3 queries per successful attack, outperforming six baselines and demonstrating strong cross-dataset and cross-model transferability. These results highlight the framework’s effectiveness and the imperative for defense mechanisms to account for continuously evolving attacker knowledge.

Abstract

The widespread deployment of Large Language Models (LLMs) as public-facing web services and APIs has made their security a core concern for the web ecosystem. Jailbreak attacks, as one of the significant threats to LLMs, have recently attracted extensive research. In this paper, we reveal a jailbreak strategy which can effectively evade current defense strategies. It can extract valuable information from failed or partially successful attack attempts and contains self-evolution from attack interactions, resulting in sufficient strategy diversity and adaptability. Inspired by continuous learning and modular design principles, we propose ASTRA, a jailbreak framework that autonomously discovers, retrieves, and evolves attack strategies to achieve more efficient and adaptive attacks. To enable this autonomous evolution, we design a closed-loop "attack-evaluate-distill-reuse" core mechanism that not only generates attack prompts but also automatically distills and generalizes reusable attack strategies from every interaction. To systematically accumulate and apply this attack knowledge, we introduce a three-tier strategy library that categorizes strategies into Effective, Promising, and Ineffective based on their performance scores. The strategy library not only provides precise guidance for attack generation but also possesses exceptional extensibility and transferability. We conduct extensive experiments under a black-box setting, and the results show that ASTRA achieves an average Attack Success Rate (ASR) of 82.7%, significantly outperforming baselines.

Figures (4)

• Figure 1: Comparison of attack pipelines. While previous attacks often rely on "templates", ASTRA extracts and accumulates "strategies" to generate novel attacks.
• Figure 2: Overview of our ASTRA framework. ASTRA is an automated jailbreaking framework that operates in a closed loop to continuously evolve attack strategies. An Attack Designer generates prompts based on accumulated strategies, a Judge Model evaluates the outcome, and a Strategy Extractor distills the interaction into a new and reusable strategy. This strategy is then stored and categorized, allowing the framework to learn from both successful and failed attempts to guide future attacks.
• Figure 3: Evolution of the strategy libraries in ASTRA. As the number of attack queries increases, the counts of Effective, Promising, and Ineffective strategies grow steadily, providing intuitive evidence of the framework’s continuous-learning capability.
• Figure 4: Attack success rate of jailbreak strategies generated by GLM-4.5 on HarmBench when transferred to AdvBench-50.