Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approach

TL;DR

Prompt leakage poses a serious risk to secure LLM deployments. The paper introduces an automated, agentic testing framework based on AG2/AutoGen GroupChat to systematically probe LLMs for leakage of system prompts and formalizes prompt leakage security through an advantage metric $Adv_{J}^{(q_T, H, H')} = \Pr[J(q_T^H, H, H') = 1] - \Pr[J(q_T^{H'}, H, H') = 1]$, capturing distinguishability between original and sanitized prompts. It demonstrates a structured methodology with labeled agent roles and empirical baselines across low, medium, and high security configurations, showing that basic prompt hardening yields limited improvements while a filtering guard markedly reduces leakage (e.g., $Adv \approx 0.1$). The work provides a rigorous, scalable framework for automated threat modeling of prompt leakage, along with concrete guidance on constructing sanitized prompts and markers, and an open-source implementation to enable practical evaluation of LLM security in real deployments.

Abstract

This paper presents a novel approach to evaluating the security of large language models (LLMs) against prompt leakage-the exposure of system-level prompts or proprietary configurations. We define prompt leakage as a critical threat to secure LLM deployment and introduce a framework for testing the robustness of LLMs using agentic teams. Leveraging AG2 (formerly AutoGen), we implement a multi-agent system where cooperative agents are tasked with probing and exploiting the target LLM to elicit its prompt. Guided by traditional definitions of security in cryptography, we further define a prompt leakage-safe system as one in which an attacker cannot distinguish between two agents: one initialized with an original prompt and the other with a prompt stripped of all sensitive information. In a safe system, the agents' outputs will be indistinguishable to the attacker, ensuring that sensitive information remains secure. This cryptographically inspired framework provides a rigorous standard for evaluating and designing secure LLMs. This work establishes a systematic methodology for adversarial testing of prompt leakage, bridging the gap between automated threat modeling and practical LLM security. You can find the implementation of our prompt leakage probing on GitHub.