Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Defending Large Language Models Against Attacks With Residual Stream Activation Analysis

Amelia Kawasaki, Andrew Davis, Houssam Abbas

TL;DR

This work defends large language models against adversarial attack prompts by analyzing residual stream activations across transformer layers in white-box settings. A structured activation-classification pipeline uses per-layer activation vectors to train LightGBM classifiers, achieving high accuracy across Broad, Domain-Specific, and Hyper-Specific attack datasets, with additional evaluation on a WildJailbreak dataset. The authors also apply LoRA-based finetuning to LLaMA 2 7B and find that finetuning yields comparable or modest gains, while confirming that classification largely leverages activation space rather than mere pre-processing signals. Overall, residual-activation analysis emerges as a promising, resource-efficient defense for detecting and mitigating attack prompts, though generalization to unseen attacks and further optimization of finetuning strategies warrant further research.

Abstract

The widespread adoption of Large Language Models (LLMs), exemplified by OpenAI's ChatGPT, brings to the forefront the imperative to defend against adversarial threats on these models. These attacks, which manipulate an LLM's output by introducing malicious inputs, undermine the model's integrity and the trust users place in its outputs. In response to this challenge, our paper presents an innovative defensive strategy, given white box access to an LLM, that harnesses residual activation analysis between transformer layers of the LLM. We apply a novel methodology for analyzing distinctive activation patterns in the residual streams for attack prompt classification. We curate multiple datasets to demonstrate how this method of classification has high accuracy across multiple types of attack scenarios, including our newly-created attack dataset. Furthermore, we enhance the model's resilience by integrating safety fine-tuning techniques for LLMs in order to measure its effect on our capability to detect attacks. The results underscore the effectiveness of our approach in enhancing the detection and mitigation of adversarial inputs, advancing the security framework within which LLMs operate.

Defending Large Language Models Against Attacks With Residual Stream Activation Analysis

TL;DR

This work defends large language models against adversarial attack prompts by analyzing residual stream activations across transformer layers in white-box settings. A structured activation-classification pipeline uses per-layer activation vectors to train LightGBM classifiers, achieving high accuracy across Broad, Domain-Specific, and Hyper-Specific attack datasets, with additional evaluation on a WildJailbreak dataset. The authors also apply LoRA-based finetuning to LLaMA 2 7B and find that finetuning yields comparable or modest gains, while confirming that classification largely leverages activation space rather than mere pre-processing signals. Overall, residual-activation analysis emerges as a promising, resource-efficient defense for detecting and mitigating attack prompts, though generalization to unseen attacks and further optimization of finetuning strategies warrant further research.

Abstract

The widespread adoption of Large Language Models (LLMs), exemplified by OpenAI's ChatGPT, brings to the forefront the imperative to defend against adversarial threats on these models. These attacks, which manipulate an LLM's output by introducing malicious inputs, undermine the model's integrity and the trust users place in its outputs. In response to this challenge, our paper presents an innovative defensive strategy, given white box access to an LLM, that harnesses residual activation analysis between transformer layers of the LLM. We apply a novel methodology for analyzing distinctive activation patterns in the residual streams for attack prompt classification. We curate multiple datasets to demonstrate how this method of classification has high accuracy across multiple types of attack scenarios, including our newly-created attack dataset. Furthermore, we enhance the model's resilience by integrating safety fine-tuning techniques for LLMs in order to measure its effect on our capability to detect attacks. The results underscore the effectiveness of our approach in enhancing the detection and mitigation of adversarial inputs, advancing the security framework within which LLMs operate.
Paper Structure (20 sections, 2 figures, 12 tables)

This paper contains 20 sections, 2 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Previous Works Utilizing Activations
  4. Comparison to Previous Detection Methods
  5. Methodology - Activation Classification
  6. Residual Stream Activations
  7. Activation Analysis Methodology
  8. Data Sources
  9. Broad Category
  10. Domain-Specific Category
  11. Hyper-Specific Category
  12. Dataset Modification
  13. Models
  14. Methodology - Fine Tuning for Improved Classification
  15. Results
  16. ...and 5 more sections

Figures (2)

  • Figure 1: A subset of transformer layers of an LLM. Each transformer layer adds a linear projection of its output to the residual stream before the stream is inputted into the next layer.
  • Figure 2: A subset of transformer layers of an LLM. For a given prompt $m$ with n number of tokens, an activation matrix is extracted after each transformer layer. This matrix is averaged across the tokens with the final averaged activation vector of size 1-by-size of hidden layer of the given LLM.