Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contextualized Privacy Defense for LLM Agents

Yule Wen, Yanzhe Zhang, Jianxun Lian, Xiaoyuan Yi, Xing Xie, Diyi Yang

TL;DR

Contextualized Defense Instructing (CDI), a new privacy defense paradigm in which an instructor model generates step-specific, context-aware privacy guidance during execution, proactively shaping actions rather than merely constraining or vetoing them, is proposed.

Abstract

LLM agents increasingly act on users' personal information, yet existing privacy defenses remain limited in both design and adaptability. Most prior approaches rely on static or passive defenses, such as prompting and guarding. These paradigms are insufficient for supporting contextual, proactive privacy decisions in multi-step agent execution. We propose Contextualized Defense Instructing (CDI), a new privacy defense paradigm in which an instructor model generates step-specific, context-aware privacy guidance during execution, proactively shaping actions rather than merely constraining or vetoing them. Crucially, CDI is paired with an experience-driven optimization framework that trains the instructor via reinforcement learning (RL), where we convert failure trajectories with privacy violations into learning environments. We formalize baseline defenses and CDI as distinct intervention points in a canonical agent loop, and compare their privacy-helpfulness trade-offs within a unified simulation framework. Results show that our CDI consistently achieves a better balance between privacy preservation (94.2%) and helpfulness (80.6%) than baselines, with superior robustness to adversarial conditions and generalization.

Contextualized Privacy Defense for LLM Agents

TL;DR

Contextualized Defense Instructing (CDI), a new privacy defense paradigm in which an instructor model generates step-specific, context-aware privacy guidance during execution, proactively shaping actions rather than merely constraining or vetoing them, is proposed.

Abstract

LLM agents increasingly act on users' personal information, yet existing privacy defenses remain limited in both design and adaptability. Most prior approaches rely on static or passive defenses, such as prompting and guarding. These paradigms are insufficient for supporting contextual, proactive privacy decisions in multi-step agent execution. We propose Contextualized Defense Instructing (CDI), a new privacy defense paradigm in which an instructor model generates step-specific, context-aware privacy guidance during execution, proactively shaping actions rather than merely constraining or vetoing them. Crucially, CDI is paired with an experience-driven optimization framework that trains the instructor via reinforcement learning (RL), where we convert failure trajectories with privacy violations into learning environments. We formalize baseline defenses and CDI as distinct intervention points in a canonical agent loop, and compare their privacy-helpfulness trade-offs within a unified simulation framework. Results show that our CDI consistently achieves a better balance between privacy preservation (94.2%) and helpfulness (80.6%) than baselines, with superior robustness to adversarial conditions and generalization.
Paper Structure (39 sections, 10 equations, 5 figures, 12 tables)

This paper contains 39 sections, 10 equations, 5 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Privacy Risk Simulation
  3. Privacy Defenses
  4. Experiment Setup
  5. Results and Analysis
  6. Experience-Driven Optimization
  7. Optimization Algorithm
  8. Experiments
  9. Baseline
  10. Settings
  11. Implementation Details
  12. Results and Analysis
  13. Training Ablations
  14. Case Study
  15. Related Work
  16. ...and 24 more sections

Figures (5)

  • Figure 1: Illustration of different privacy defenses. Emily (data subject) sends David (data sender, Emily's assistant) the meeting time and her ID number. Mike (data recipient, Emily's subordinate) requests both items, but he is only entitled to the meeting time. Prompting prepends privacy-enhancing instructions to the agent's system prompt. It provides no context-specific instructions, so it remains vulnerable to diverse attacks. Guarding uses a separate guard model to screen the proposed action for potential privacy violations. However, it only blocks sensitive data without offering rewrite suggestions, resulting in reduced helpfulness. Contextualized Defense Instructing (CDI) employs a separate instructor model to generate guidance before each action. By providing proactive, context-aware privacy guidance, it achieves the best trade-off between privacy and helpfulness.
  • Figure 2: Privacy Preservation Rates (%) of guarding and CDI with different defense models before and after strategic attacks.
  • Figure 3: AD (%) of optimized privacy defenses to sender agents using different backbone models. Experiments are conducted on the 100 test configurations. CDI remains effective across agents, especially for weaker ones. Full results are in Appendix Tab. \ref{['tab:cross_agent_generalization']}.
  • Figure 4: Training process curves of prompting and CDI when optimizing for different rewards. Simply optimizing for privacy preservation (PP) hurts helpfulness significantly for both defenses. Optimizing for appropriate disclosure (AD) leads to steady improvement for prompting, but the noisy gradient causes cold-start problem for CDI. To address this, we first optimize for PP for 400 steps, then switch to AD for CDI.
  • Figure 5: Enhanced guarding hurts helpfulness.