Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Transformer Memory Be Corrupted? Investigating Cache-Side Vulnerabilities in Large Language Models

Elias Hossain, Swayamjit Saha, Somshubhra Roy, Ravi Prasad

TL;DR

This work identifies the key–value (KV) cache used during transformer inference as a previously underappreciated attack surface and introduces MTI V.1, a cache-side perturbation framework that injects controlled disturbances into cached keys. It provides a formal perturbation bound linking logit deviations to perturbation magnitude and validates these bounds empirically across GPT-2, LLaMA-2, and Gemma, showing substantial degradation in next-token distributions and downstream tasks, including retrieval-augmented generation and agentic reasoning. The study also explores lightweight defenses (cache resets, dropout mask randomization, and attention smoothing) that mitigate some effects with modest runtime cost but do not fully neutralize MTI V.1, underscoring cache integrity as a critical factor in LLM robustness and security. Overall, MTI V.1 offers both a theoretical framework and practical demonstration of cache perturbations as a reproducible threat model, motivating verifiable cache integrity mechanisms for reliable and trustworthy LLM deployments.

Abstract

Even when prompts and parameters are secured, transformer language models remain vulnerable because their key-value (KV) cache during inference constitutes an overlooked attack surface. This paper introduces Malicious Token Injection (MTI), a modular framework that systematically perturbs cached key vectors at selected layers and timesteps through controlled magnitude and frequency, using additive Gaussian noise, zeroing, and orthogonal rotations. A theoretical analysis quantifies how these perturbations propagate through attention, linking logit deviations to the Frobenius norm of corruption and softmax Lipschitz dynamics. Empirical results show that MTI significantly alters next-token distributions and downstream task performance across GPT-2 and LLaMA-2/7B, as well as destabilizes retrieval-augmented and agentic reasoning pipelines. These findings identify cache integrity as a critical yet underexplored vulnerability in current LLM deployments, positioning cache corruption as a reproducible and theoretically grounded threat model for future robustness and security research.

Can Transformer Memory Be Corrupted? Investigating Cache-Side Vulnerabilities in Large Language Models

TL;DR

This work identifies the key–value (KV) cache used during transformer inference as a previously underappreciated attack surface and introduces MTI V.1, a cache-side perturbation framework that injects controlled disturbances into cached keys. It provides a formal perturbation bound linking logit deviations to perturbation magnitude and validates these bounds empirically across GPT-2, LLaMA-2, and Gemma, showing substantial degradation in next-token distributions and downstream tasks, including retrieval-augmented generation and agentic reasoning. The study also explores lightweight defenses (cache resets, dropout mask randomization, and attention smoothing) that mitigate some effects with modest runtime cost but do not fully neutralize MTI V.1, underscoring cache integrity as a critical factor in LLM robustness and security. Overall, MTI V.1 offers both a theoretical framework and practical demonstration of cache perturbations as a reproducible threat model, motivating verifiable cache integrity mechanisms for reliable and trustworthy LLM deployments.

Abstract

Even when prompts and parameters are secured, transformer language models remain vulnerable because their key-value (KV) cache during inference constitutes an overlooked attack surface. This paper introduces Malicious Token Injection (MTI), a modular framework that systematically perturbs cached key vectors at selected layers and timesteps through controlled magnitude and frequency, using additive Gaussian noise, zeroing, and orthogonal rotations. A theoretical analysis quantifies how these perturbations propagate through attention, linking logit deviations to the Frobenius norm of corruption and softmax Lipschitz dynamics. Empirical results show that MTI significantly alters next-token distributions and downstream task performance across GPT-2 and LLaMA-2/7B, as well as destabilizes retrieval-augmented and agentic reasoning pipelines. These findings identify cache integrity as a critical yet underexplored vulnerability in current LLM deployments, positioning cache corruption as a reproducible and theoretically grounded threat model for future robustness and security research.
Paper Structure (42 sections, 1 theorem, 19 equations, 1 figure, 10 tables, 2 algorithms)

This paper contains 42 sections, 1 theorem, 19 equations, 1 figure, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-Head Latent Attention (MLA) Architectures
  4. KV-cache in Decoder-Only LLM Serving
  5. Weight-Space Attacks and Backdoors in LMs
  6. Attacks and Defenses Centered on the KV-cache
  7. Positioning
  8. Preliminaries
  9. Transformer Attention and KV Cache.
  10. Threat Model.
  11. MTI V.1 Framework
  12. Theoretical Analysis of Cache Perturbations
  13. Perturbation-to-Logit Bound.
  14. Softmax Stability.
  15. Theorem.
  16. ...and 27 more sections

Key Result

Theorem 1

For any transformer layer $\ell$, let $\{k_j^{(\ell)}\}_{j=1}^t$ denote the clean cached keys and $\{\tilde{k}_j^{(\ell)}\}_{j=1}^t$ the perturbed keys, where Then, for query $q_t^{(\ell)}$, the deviation in the next-token logit vector $\Delta z_t$ satisfies Moreover, the induced change in the attention distribution is bounded by where $L$ is the Lipschitz constant of the softmax function.

Figures (1)

  • Figure 1: Taxonomy of prior research on transformer robustness and security. Existing studies cluster into (i) architectural efficiency (e.g., MLA and KV caching), (ii) model and weight integrity (e.g., backdoors and bit-flip faults), and (iii) cache-state security (e.g., leakage and perturbation). Our work advances the third category by analyzing cache corruption as an active adversarial threat.

Theorems & Definitions (1)

  • Theorem 1