Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models
Zhiyuan Yu, Xiaogeng Liu, Shunning Liang, Zach Cameron, Chaowei Xiao, Ning Zhang
TL;DR
This paper investigates jailbreaking prompts for large language models by (i) systemizing 448 in-the-wild jailbreak prompts into 5 categories and 10 patterns, (ii) empirically evaluating their effectiveness on GPT-3.5, GPT-4, and PaLM-2 using human-annotated metrics EMH and JSR, (iii) conducting a 92-participant human study to observe how novices and experts craft prompts with and without AI assistance, and (iv) proposing an automatic jailbreak prompt generation framework that iteratively mutates prompts and uses feedback to maximize jailbreak efficacy. The work reveals that longer, semantically meaningful prompts, especially those employing Virtual AI Simulation and Hybrid Strategies, reliably bypass defenses, and identifies universal prompts that trigger jailbreaks across models. It also demonstrates that humans—even without AI aid—can craft effective prompts, and that AI collaboration can automate substantial portions of the process, informing both attack understanding and defense design. These insights have implications for improving LLM safety, prompt auditing, and defense strategies in real-world deployments, where adaptive jailbreak tactics pose ongoing risks. All metrics and data are framed to quantify worst-case harm and overall jailbreak propensity, highlighting practical paths to strengthen alignment and safety controls.
Abstract
Recent advancements in generative AI have enabled ubiquitous access to large language models (LLMs). Empowered by their exceptional capabilities to understand and generate human-like text, these models are being increasingly integrated into our society. At the same time, there are also concerns on the potential misuse of this powerful technology, prompting defensive measures from service providers. To overcome such protection, jailbreaking prompts have recently emerged as one of the most effective mechanisms to circumvent security restrictions and elicit harmful content originally designed to be prohibited. Due to the rapid development of LLMs and their ease of access via natural languages, the frontline of jailbreak prompts is largely seen in online forums and among hobbyists. To gain a better understanding of the threat landscape of semantically meaningful jailbreak prompts, we systemized existing prompts and measured their jailbreak effectiveness empirically. Further, we conducted a user study involving 92 participants with diverse backgrounds to unveil the process of manually creating jailbreak prompts. We observed that users often succeeded in jailbreak prompts generation regardless of their expertise in LLMs. Building on the insights from the user study, we also developed a system using AI as the assistant to automate the process of jailbreak prompt generation.