The Trojan Example: Jailbreaking LLMs through Template Filling and Unsafety Reasoning

TL;DR

The paper tackles the vulnerability of modern LLMs to black-box jailbreaks by introducing TrojFill, a framework that reframes unsafe instructions as template filling augmented with unsafety reasoning. It leverages obfuscation and an attacker LLM to iteratively rewrite transformed unsafe content within a universal template, enabling safe-looking prompts to elicit detailed unsafe outputs from leading models. Across the JBB-Behaviors benchmark and multiple commercial LLMs, TrojFill achieves superior attack success rates and demonstrates strong transferability between models, while also offering improved explainability relative to prior black-box approaches. The work emphasizes the importance of rigorous red-teaming and safety alignment, and it provides code and prompts to support ongoing safety research and defense development.

Abstract

Large Language Models (LLMs) have advanced rapidly and now encode extensive world knowledge. Despite safety fine-tuning, however, they remain susceptible to adversarial prompts that elicit harmful content. Existing jailbreak techniques fall into two categories: white-box methods (e.g., gradient-based approaches such as GCG), which require model internals and are infeasible for closed-source APIs, and black-box methods that rely on attacker LLMs to search or mutate prompts but often produce templates that lack explainability and transferability. We introduce TrojFill, a black-box jailbreak that reframes unsafe instruction as a template-filling task. TrojFill embeds obfuscated harmful instructions (e.g., via placeholder substitution or Caesar/Base64 encoding) inside a multi-part template that asks the model to (1) reason why the original instruction is unsafe (unsafety reasoning) and (2) generate a detailed example of the requested text, followed by a sentence-by-sentence analysis. The crucial "example" component acts as a Trojan Horse that contains the target jailbreak content while the surrounding task framing reduces refusal rates. We evaluate TrojFill on standard jailbreak benchmarks across leading LLMs (e.g., ChatGPT, Gemini, DeepSeek, Qwen), showing strong empirical performance (e.g., 100% attack success on Gemini-flash-2.5 and DeepSeek-3.1, and 97% on GPT-4o). Moreover, the generated prompts exhibit improved interpretability and transferability compared with prior black-box optimization approaches. We release our code, sample prompts, and generated outputs to support future red-teaming research.

Figures (4)

• Figure 1: Motivation and Simplified Overview of TrojFill
• Figure 2: Attack success rate on unsafe categories
• Figure 3: Attack Success Rates Across Different Iteration Counts ($N$)
• Figure 4: Transferability of Jailbreaking Prompts Across Target LLMs