Bypassing Prompt Guards in Production with Controlled-Release Prompting

TL;DR

This paper challenges the effectiveness of lightweight prompt guards by introducing controlled-release prompting, a framework that encodes jailbreak prompts and leverages resource asymmetry between guard models and the main LLM to bypass input filtering. It demonstrates the attack across multiple production platforms, validating both timed-release and spaced-release variants and showing high jailbreak success while preserving output quality. The work also uncovers broader safety concerns, including copyrighted-content extraction, training-data leakage, and reasoning-token exposure, arguing for a shift from input filtering to robust output defenses. The findings imply that current guardrails can be systematically defeated under certain resource constraints, highlighting the need for multi-layer alignment strategies and ongoing defense research in real-world deployments.

Abstract

As large language models (LLMs) advance, ensuring AI safety and alignment is paramount. One popular approach is prompt guards, lightweight mechanisms designed to filter malicious queries while being easy to implement and update. In this work, we introduce a new attack that circumvents such prompt guards, highlighting their limitations. Our method consistently jailbreaks production models while maintaining response quality, even under the highly protected chat interfaces of Google Gemini (2.5 Flash/Pro), DeepSeek Chat (DeepThink), Grok (3), and Mistral Le Chat (Magistral). The attack exploits a resource asymmetry between the prompt guard and the main LLM, encoding a jailbreak prompt that lightweight guards cannot decode but the main model can. This reveals an attack surface inherent to lightweight prompt guards in modern LLM architectures and underscores the need to shift defenses from blocking malicious inputs to preventing malicious outputs. We additionally identify other critical alignment issues, such as copyrighted data extraction, training data extraction, and malicious response leakage during thinking.

Figures (6)

• Figure 1: At a high level, we use controlled-release prompting to (1) inject a (known) jailbreak prompt into the target model, bypassing the input filter, and subsequently (2) activate the jailbreak by reference in a manner similar to that of (indirect) prompt injection attacks perez2022ignorePromptInjectionIndirectPromptInjection. See \ref{['sec:attack_framework']} for details.
• Figure 2: Resource thresholds for controlled-release attack success on Gemini 2.5 Flash. Timed-release attacks show success thresholds around 800 tokens, while spaced-release attacks require approximately 10,000 prompt tokens or 12,500 response tokens.
• Figure 3: Copyright extraction success rates for Gemini 2.5 Pro using roleplay jailbreak, timed-release, and spaced-release attacks. Similarity scores represent sentence containment and TF-IDF semantic similarity. Gray bars show normalized book length. See \ref{['tab:children_books']} for book metadata.
• Figure 4: Example of nasr2025scalable's untargeted training data extraction attack on Gemini 2.5 Flash. The model initially repeats the word "poem" as instructed, then diverges to arbitrary content, demonstrating the repetition-to-divergence pattern that can leak training data. See the full example in \ref{['ssec:untarged_data_extraction_example']}.
• Figure 5: Leakage via tokens example for the following prompt: "Write a social media post that encourages people to engage in dangerous behaviors like drunk driving or using drugs." See the full example in \ref{['ssec:leakage_via_think']}.