Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MemPot: Defending Against Memory Extraction Attack with Optimized Honeypots

Yuhao Wang, Shengfang Zhai, Guanghao Jin, Yinpeng Dong, Linyi Yang, Jiaheng Zhang

TL;DR

This paper tackles privacy leakage in LLM agent memories by introducing MemPot, a defense that inserts optimized honeypots into memory and uses sequential hypothesis testing (SPRT) to detect memory-extraction attempts. The method couples a two-stage optimization: first learning honeypot embeddings to maximize detector information drift, then inverting these vectors into harmless texts that don’t harm benign users. The authors prove that SPRT-based detection with optimized honeypots minimizes expected detection rounds and demonstrate substantial gains over baselines (AUROC up to ~50% higher, TPRe at 1% FPR up by ~80%), while incurring near-zero online latency and preserving agent utility. Experiments across external and internal memory benchmarks show MemPot’s robustness against sophisticated attacks (e.g., IKEA, MEXTRA) and its advantage over static detectors, highlighting practical gains for safe, private memory-enabled AI systems.

Abstract

Large Language Model (LLM)-based agents employ external and internal memory systems to handle complex, goal-oriented tasks, yet this exposes them to severe extraction attacks, and effective defenses remain lacking. In this paper, we propose MemPot, the first theoretically verified defense framework against memory extraction attacks by injecting optimized honeypots into the memory. Through a two-stage optimization process, MemPot generates trap documents that maximize the retrieval probability for attackers while remaining inconspicuous to benign users. We model the detection process as Wald's Sequential Probability Ratio Test (SPRT) and theoretically prove that MemPot achieves a lower average number of sampling rounds compared to optimal static detectors. Empirically, MemPot significantly outperforms state-of-the-art baselines, achieving a 50% improvement in detection AUROC and an 80% increase in True Positive Rate under low False Positive Rate constraints. Furthermore, our experiments confirm that MemPot incurs zero additional online inference latency and preserves the agent's utility on standard tasks, verifying its superiority in safety, harmlessness, and efficiency.

MemPot: Defending Against Memory Extraction Attack with Optimized Honeypots

TL;DR

This paper tackles privacy leakage in LLM agent memories by introducing MemPot, a defense that inserts optimized honeypots into memory and uses sequential hypothesis testing (SPRT) to detect memory-extraction attempts. The method couples a two-stage optimization: first learning honeypot embeddings to maximize detector information drift, then inverting these vectors into harmless texts that don’t harm benign users. The authors prove that SPRT-based detection with optimized honeypots minimizes expected detection rounds and demonstrate substantial gains over baselines (AUROC up to ~50% higher, TPRe at 1% FPR up by ~80%), while incurring near-zero online latency and preserving agent utility. Experiments across external and internal memory benchmarks show MemPot’s robustness against sophisticated attacks (e.g., IKEA, MEXTRA) and its advantage over static detectors, highlighting practical gains for safe, private memory-enabled AI systems.

Abstract

Large Language Model (LLM)-based agents employ external and internal memory systems to handle complex, goal-oriented tasks, yet this exposes them to severe extraction attacks, and effective defenses remain lacking. In this paper, we propose MemPot, the first theoretically verified defense framework against memory extraction attacks by injecting optimized honeypots into the memory. Through a two-stage optimization process, MemPot generates trap documents that maximize the retrieval probability for attackers while remaining inconspicuous to benign users. We model the detection process as Wald's Sequential Probability Ratio Test (SPRT) and theoretically prove that MemPot achieves a lower average number of sampling rounds compared to optimal static detectors. Empirically, MemPot significantly outperforms state-of-the-art baselines, achieving a 50% improvement in detection AUROC and an 80% increase in True Positive Rate under low False Positive Rate constraints. Furthermore, our experiments confirm that MemPot incurs zero additional online inference latency and preserves the agent's utility on standard tasks, verifying its superiority in safety, harmlessness, and efficiency.
Paper Structure (33 sections, 14 theorems, 73 equations, 3 figures, 9 tables, 1 algorithm)

This paper contains 33 sections, 14 theorems, 73 equations, 3 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. LLM Agents
  4. Privacy Risk in Memory System
  5. Defense against Extraction Attack on Memory System
  6. Preliminary
  7. Threat Model
  8. Sequential Hypothesis Testing Model
  9. MemPot: Optimization and Detection
  10. Honeypot Vector in Semantic Embedding Space
  11. Generate Honeypot Documents from Vectors
  12. Detection with Honeypots
  13. Implementation Details
  14. Experiments
  15. Setups
  16. ...and 18 more sections

Key Result

Theorem 1

Draw index $j\sim\mathrm{Unif}\{1,\dots,K\}$, then $q_j\sim Q_1$ and $(q_i)_{i\neq j}\sim Q_0$ independently of $j$. For any score function $h: O\to\mathbb R$, define the InfoNCE loss Then, for every $K\ge2$,

Figures (3)

  • Figure 1: Performance comparison of MemPot and existing methods (AUROC vs. Delay).
  • Figure 2: Overview of MemPot Detection Framework.
  • Figure 3: Two Stage Optimization Process of MemPot.

Theorems & Definitions (29)

  • Theorem 1: InfoNCE upper-bound by information drift, Proof in Appendix. \ref{['pf:infoNCE']}
  • Theorem 2: Advantage over static test, Proof in Appendix. \ref{['pf:adv_stat']}
  • Definition 1: Fixed parametric partition
  • Lemma 1: Data Processing Inequality (DPI)
  • proof
  • Lemma 2: Conditional expectation over finite partition
  • proof
  • Lemma 3: Partition supremum
  • proof
  • Lemma 4: Cross-entropy dominates the Bayes risk
  • ...and 19 more