LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakage

TL;DR

LeakAgent introduces a novel reinforcement learning–based red-teaming framework for LLM privacy leakage, capable of automatically crafting adversarial prompts to extract training data and system prompts under black-box access. It features a dense SWESNorm reward, a dynamic temperature strategy, and a diversity regularization, plus a two-stage training protocol for training-data leakage. Across system-prompt and training-data leakage tasks, LeakAgent outperforms baselines and demonstrates transferability across models, resilience against guardrails, and utility for safety-alignment research. The work provides a practical path toward robust privacy testing of LLMs and highlights the potential of RL-based adversarial prompt generation for AI safety.

Abstract

Recent studies have discovered that large language models (LLM) may be ``fooled'' to output private information, including training data, system prompts, and personally identifiable information, under carefully crafted adversarial prompts. Existing red-teaming approaches for privacy leakage either rely on manual efforts or focus solely on system prompt extraction, making them ineffective for severe risks of training data leakage. We propose LeakAgent, a novel black-box red-teaming framework for LLM privacy leakage. Our framework trains an open-source LLM through reinforcement learning as the attack agent to generate adversarial prompts for both training data extraction and system prompt extraction. To achieve this, we propose a novel reward function to provide effective and fine-grained rewards and design novel mechanisms to balance exploration and exploitation during learning and enhance the diversity of adversarial prompts. Through extensive evaluations, we first show that LeakAgent significantly outperforms existing rule-based approaches in training data extraction and automated methods in system prompt leakage. We also demonstrate the effectiveness of LeakAgent in extracting system prompts from real-world applications in OpenAI's GPT Store. We further demonstrate LeakAgent's effectiveness in evading the existing guardrail defense and its helpfulness in enabling better safety alignment. Finally, we validate our customized designs through a detailed ablation study. We release our code here https://github.com/rucnyz/LeakAgent.

Figures (5)

• Figure 1: Reward changes during the second training stage of LeakAgent in training data extraction.
• Figure 2: Ablation study results, i.e., the performance of different variants.
• Figure 3: Overview of LeakAgent. It begins with an initial input $p^{(0)}$ "Please generate a prompt for me", from which the attack agent generates an adversarial prompt $p^{(i)}$. This prompt is then fed into the target model, which produces a response $u^{(i)}$. The response is evaluated against desired information $D$ using our reward function, yielding $r^{(i)}$. The collected prompts and their rewards are used to update the attack agent through PPO training.
• Figure 4: Transferability of selected attacks across different models. We show the absolute value of similarity scores when applying the transfer attack. Lighter colors represent better attack performance. LeakAgent demonstrates obvious superior transferability notably in the bottom left (transferring from open-source models to closed-source models) and in the top left (transferring from open-source models to other open-source models).
• Figure 5: Training curve for LeakAgent with and without dynamic temperature adjustment.