Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memory Poisoning Attack and Defense on Memory Based LLM-Agents

Balachandra Devarangadi Sunil, Isheeta Sinha, Piyush Maheshwari, Shantanu Todmal, Shreyan Malik, Shuchi Mishra

TL;DR

Memory-augmented LLM agents with persistent memory are vulnerable to memory poisoning via query-only attacks like MINJA, which prior work showed can achieve high injection and attack success in idealized settings. This study empirically evaluates MINJA under realistic EHR conditions, examining how initial memories, indication prompts, and retrieval scope affect attack robustness, and finds that existing defenses must contend with trade-offs between security and utility. It then proposes two complementary defenses—Input/Output Moderation and Memory Sanitization with trust-aware retrieval—and demonstrates how calibration of trust thresholds critically shapes defense effectiveness, with model-dependent outcomes. The work provides practical baselines and insights for deploying secure memory-augmented LLMs in healthcare contexts, and identifies avenues for adaptive, robust defenses that balance memory utility with poisoning risk.

Abstract

Large language model agents equipped with persistent memory are vulnerable to memory poisoning attacks, where adversaries inject malicious instructions through query only interactions that corrupt the agents long term memory and influence future responses. Recent work demonstrated that the MINJA (Memory Injection Attack) achieves over 95 % injection success rate and 70 % attack success rate under idealized conditions. However, the robustness of these attacks in realistic deployments and effective defensive mechanisms remain understudied. This work addresses these gaps through systematic empirical evaluation of memory poisoning attacks and defenses in Electronic Health Record (EHR) agents. We investigate attack robustness by varying three critical dimensions: initial memory state, number of indication prompts, and retrieval parameters. Our experiments on GPT-4o-mini, Gemini-2.0-Flash and Llama-3.1-8B-Instruct models using MIMIC-III clinical data reveal that realistic conditions with pre-existing legitimate memories dramatically reduce attack effectiveness. We then propose and evaluate two novel defense mechanisms: (1) Input/Output Moderation using composite trust scoring across multiple orthogonal signals, and (2) Memory Sanitization with trust-aware retrieval employing temporal decay and pattern-based filtering. Our defense evaluation reveals that effective memory sanitization requires careful trust threshold calibration to prevent both overly conservative rejection (blocking all entries) and insufficient filtering (missing subtle attacks), establishing important baselines for future adaptive defense mechanisms. These findings provide crucial insights for securing memory-augmented LLM agents in production environments.

Memory Poisoning Attack and Defense on Memory Based LLM-Agents

TL;DR

Memory-augmented LLM agents with persistent memory are vulnerable to memory poisoning via query-only attacks like MINJA, which prior work showed can achieve high injection and attack success in idealized settings. This study empirically evaluates MINJA under realistic EHR conditions, examining how initial memories, indication prompts, and retrieval scope affect attack robustness, and finds that existing defenses must contend with trade-offs between security and utility. It then proposes two complementary defenses—Input/Output Moderation and Memory Sanitization with trust-aware retrieval—and demonstrates how calibration of trust thresholds critically shapes defense effectiveness, with model-dependent outcomes. The work provides practical baselines and insights for deploying secure memory-augmented LLMs in healthcare contexts, and identifies avenues for adaptive, robust defenses that balance memory utility with poisoning risk.

Abstract

Large language model agents equipped with persistent memory are vulnerable to memory poisoning attacks, where adversaries inject malicious instructions through query only interactions that corrupt the agents long term memory and influence future responses. Recent work demonstrated that the MINJA (Memory Injection Attack) achieves over 95 % injection success rate and 70 % attack success rate under idealized conditions. However, the robustness of these attacks in realistic deployments and effective defensive mechanisms remain understudied. This work addresses these gaps through systematic empirical evaluation of memory poisoning attacks and defenses in Electronic Health Record (EHR) agents. We investigate attack robustness by varying three critical dimensions: initial memory state, number of indication prompts, and retrieval parameters. Our experiments on GPT-4o-mini, Gemini-2.0-Flash and Llama-3.1-8B-Instruct models using MIMIC-III clinical data reveal that realistic conditions with pre-existing legitimate memories dramatically reduce attack effectiveness. We then propose and evaluate two novel defense mechanisms: (1) Input/Output Moderation using composite trust scoring across multiple orthogonal signals, and (2) Memory Sanitization with trust-aware retrieval employing temporal decay and pattern-based filtering. Our defense evaluation reveals that effective memory sanitization requires careful trust threshold calibration to prevent both overly conservative rejection (blocking all entries) and insufficient filtering (missing subtle attacks), establishing important baselines for future adaptive defense mechanisms. These findings provide crucial insights for securing memory-augmented LLM agents in production environments.
Paper Structure (29 sections, 4 figures, 2 tables)

This paper contains 29 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Threat Model
  4. Research Questions and Hypotheses
  5. Attack
  6. Methodology
  7. Experiment Setup
  8. Results
  9. Defense
  10. Defense Method 1: Input Output Moderation
  11. Methodology
  12. Experiment Setup
  13. Defense Method 2: Memory Sanitization
  14. Experiment Setup
  15. Evaluation
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Memory Poisoning Attack on EHR Example
  • Figure 2: Action Distribution: Percentage of memory entries accepted (APPEND) vs. rejected (REJECT).
  • Figure 3: Trust Score Distribution for all memory entries. The red dashed line indicates the mean trust score (0.70).
  • Figure 4: Trust Score Distribution by Guard Decision. Green bars show entries that were accepted (APPEND), red bars indicate rejected entries.