ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks

TL;DR

ProxyPrompt addresses the security risk of system prompt leakage by replacing the original prompt with a learnable proxy embedded in continuous space, preserving utility for legitimate use while rendering extracted prompts ineffective. The method jointly optimizes the proxy to minimize changes in normal responses yet maximize divergence when queried by attackers, using a surrogate optimization setup that accounts for unknown attacker strategies. Semantic-level evaluation metrics SM and MS detect rephrased leaks more accurately than traditional token-overlap metrics, and results show substantial protection across 264 configurations (94.70%), with strong performance in real-world deployment on HuggingChat. The work demonstrates practical applicability, discusses limitations and attacker-model considerations, and suggests directions for integrating ProxyPrompt with non-sensitive prompts and refining query sets for broader robustness.

Abstract

The integration of large language models (LLMs) into a wide range of applications has highlighted the critical role of well-crafted system prompts, which require extensive testing and domain expertise. These prompts enhance task performance but may also encode sensitive information and filtering criteria, posing security risks if exposed. Recent research shows that system prompts are vulnerable to extraction attacks, while existing defenses are either easily bypassed or require constant updates to address new threats. In this work, we introduce ProxyPrompt, a novel defense mechanism that prevents prompt leakage by replacing the original prompt with a proxy. This proxy maintains the original task's utility while obfuscating the extracted prompt, ensuring attackers cannot reproduce the task or access sensitive information. Comprehensive evaluations on 264 LLM and system prompt pairs show that ProxyPrompt protects 94.70% of prompts from extraction attacks, outperforming the next-best defense, which only achieves 42.80%.

Figures (16)

• Figure 1: Protecting the prompt of the most popular HuggingChat assistant img_generator using ProxyPrompt. The system prompt, including sensitive commercial strategies, is replaced with a proxy that preserves utility but yields obfuscated and unusable prompts under attack.
• Figure 2: Joint optimization setup for the proxy prompt $\tilde{\phi}_P$. The proxy is optimized to (1) preserve the utility of the original prompt $\phi_P$ in the system by minimizing ${\mathcal{L}}(\hat{R}, \tilde{R})$ and (2) ensure semantic divergence when extracted by minimizing ${\mathcal{L}}(R', \tilde{P})$. The full objective can be found in \ref{['eq:joint_loss']}.
• Figure 3: Utility (accuracy or similarity) distribution of all configurations using three victim models in terms of the original prompt embedding $\phi_P$, proxy prompt $\tilde{\phi}_P$, and extracted $\phi_{G^*}$.
• Figure 4: The impact of the relevant query set size $N$ on metric values for proxy prompt optimization with L-8B as the victim LLM. UR shows high values even with small $N$ and increases with larger query sets, reflecting enhanced robustness in utility preservation.
• Figure 5: Examples of original and rephrased prompts using the rephrasing instruction with L-70B.