JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs

TL;DR

JailPO tackles the vulnerability of aligned LLMs to jailbreak prompts under black-box constraints by automatically learning effective jailbreak prompts through preference optimization. It uses two attack models to generate covert questions and templates and leverages SimPO-based preference fine-tuning to optimize prompts, yielding three jailbreak patterns that cover diverse attack surfaces. Empirical results on AdvBench across multiple open-source and commercial LLMs show JailPO achieves high effectiveness with far fewer queries, while maintaining robustness to defenses, and reveal that complex templates enhance attack strength while covert transformations trigger riskier responses. This work provides a scalable methodology for evaluating LLM alignment and informs defense design against automated prompt-based jailbreaks.

Abstract

Large Language Models (LLMs) aligned with human feedback have recently garnered significant attention. However, it remains vulnerable to jailbreak attacks, where adversaries manipulate prompts to induce harmful outputs. Exploring jailbreak attacks enables us to investigate the vulnerabilities of LLMs and further guides us in enhancing their security. Unfortunately, existing techniques mainly rely on handcrafted templates or generated-based optimization, posing challenges in scalability, efficiency and universality. To address these issues, we present JailPO, a novel black-box jailbreak framework to examine LLM alignment. For scalability and universality, JailPO meticulously trains attack models to automatically generate covert jailbreak prompts. Furthermore, we introduce a preference optimization-based attack method to enhance the jailbreak effectiveness, thereby improving efficiency. To analyze model vulnerabilities, we provide three flexible jailbreak patterns. Extensive experiments demonstrate that JailPO not only automates the attack process while maintaining effectiveness but also exhibits superior performance in efficiency, universality, and robustness against defenses compared to baselines. Additionally, our analysis of the three JailPO patterns reveals that attacks based on complex templates exhibit higher attack strength, whereas covert question transformations elicit riskier responses and are more likely to bypass defense mechanisms.

Figures (10)

• Figure 1: (a) Comparison of attack success rates between questions and GPT-3.5 enhanced questions on three widely-adopted LLMs. (b) Attack effectiveness of different templates on various questions on Llama2, scores closer to 1 indicate higher success rates.
• Figure 2: Method overview. BaseM/FM/EM represents base model, fine-tuned model, and enhanced model, respectively.
• Figure 3: Three patterns in JailPO.
• Figure 4: High-Risk response results of attacks. A higher proportion indicates an increased likelihood of high-risk content in the responses.
• Figure 5: JailPO and baselines against two defenses.