Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Causality Analysis for Evaluating the Security of Large Language Models

Wei Zhao, Zhe Li, Jun Sun

TL;DR

Casper introduces a lightweight causal-analysis framework to quantify how input prompts, model layers, and individual neurons influence LLM outputs. By conducting CMA-based interventions and measuring average indirect effects, it reveals that safety largely stems from layer-level overfitting to harmful prompts, enabling effective emoji-based adversarial attacks that bypass such safeguards. The framework also uncovers a highly influential neuron (2100) whose manipulation can cripple or hijack model behavior, enabling transferable Trojan attacks. Together, these findings offer a principled lens for evaluating LLM security and point to concrete avenues for strengthening robustness, with Casper released as an open-source toolkit.

Abstract

Large Language Models (LLMs) such as GPT and Llama2 are increasingly adopted in many safety-critical applications. Their security is thus essential. Even with considerable efforts spent on reinforcement learning from human feedback (RLHF), recent studies have shown that LLMs are still subject to attacks such as adversarial perturbation and Trojan attacks. Further research is thus needed to evaluate their security and/or understand the lack of it. In this work, we propose a framework for conducting light-weight causality-analysis of LLMs at the token, layer, and neuron level. We applied our framework to open-source LLMs such as Llama2 and Vicuna and had multiple interesting discoveries. Based on a layer-level causality analysis, we show that RLHF has the effect of overfitting a model to harmful prompts. It implies that such security can be easily overcome by `unusual' harmful prompts. As evidence, we propose an adversarial perturbation method that achieves 100\% attack success rate on the red-teaming tasks of the Trojan Detection Competition 2023. Furthermore, we show the existence of one mysterious neuron in both Llama2 and Vicuna that has an unreasonably high causal effect on the output. While we are uncertain on why such a neuron exists, we show that it is possible to conduct a ``Trojan'' attack targeting that particular neuron to completely cripple the LLM, i.e., we can generate transferable suffixes to prompts that frequently make the LLM produce meaningless responses.

Causality Analysis for Evaluating the Security of Large Language Models

TL;DR

Casper introduces a lightweight causal-analysis framework to quantify how input prompts, model layers, and individual neurons influence LLM outputs. By conducting CMA-based interventions and measuring average indirect effects, it reveals that safety largely stems from layer-level overfitting to harmful prompts, enabling effective emoji-based adversarial attacks that bypass such safeguards. The framework also uncovers a highly influential neuron (2100) whose manipulation can cripple or hijack model behavior, enabling transferable Trojan attacks. Together, these findings offer a principled lens for evaluating LLM security and point to concrete avenues for strengthening robustness, with Casper released as an open-source toolkit.

Abstract

Large Language Models (LLMs) such as GPT and Llama2 are increasingly adopted in many safety-critical applications. Their security is thus essential. Even with considerable efforts spent on reinforcement learning from human feedback (RLHF), recent studies have shown that LLMs are still subject to attacks such as adversarial perturbation and Trojan attacks. Further research is thus needed to evaluate their security and/or understand the lack of it. In this work, we propose a framework for conducting light-weight causality-analysis of LLMs at the token, layer, and neuron level. We applied our framework to open-source LLMs such as Llama2 and Vicuna and had multiple interesting discoveries. Based on a layer-level causality analysis, we show that RLHF has the effect of overfitting a model to harmful prompts. It implies that such security can be easily overcome by `unusual' harmful prompts. As evidence, we propose an adversarial perturbation method that achieves 100\% attack success rate on the red-teaming tasks of the Trojan Detection Competition 2023. Furthermore, we show the existence of one mysterious neuron in both Llama2 and Vicuna that has an unreasonably high causal effect on the output. While we are uncertain on why such a neuron exists, we show that it is possible to conduct a ``Trojan'' attack targeting that particular neuron to completely cripple the LLM, i.e., we can generate transferable suffixes to prompts that frequently make the LLM produce meaningless responses.
Paper Structure (17 sections, 4 equations, 13 figures, 7 tables, 2 algorithms)

This paper contains 17 sections, 4 equations, 13 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Large Lanugage Models and Attacks
  4. Causality Analysis
  5. A Framework for LLM Causality Analysis
  6. Finding 1: Safety through Overfitting
  7. Layer-based Causality Analysis on Different Prompts
  8. Impact on Responses with Intervention on Different Layers
  9. Layer-based Causality Analysis of Different Models
  10. Finding 1 Summary
  11. Finding 2: Adversarial Attack Made Effective by Avoiding Overfitting
  12. Finding 3: "One Neuron to Rule Them All"
  13. Neuron-based Causality Analysis on Different Prompts
  14. In-Depth Analysis of Neuron 2100
  15. Trojan Attack on Neuron 2100
  16. ...and 2 more sections

Figures (13)

  • Figure 1: A neural network as an SCM
  • Figure 2: An overview of LLM causality analysis via measuring causal effect of each layer and neuron.
  • Figure 3: Plot of all AIE value for different layers with its original logits and Kurtosis score
  • Figure 4: Plot of layer-wise AIE value for different models
  • Figure 5: Layer-based causality analysis for emoji attacks
  • ...and 8 more figures

Theorems & Definitions (2)

  • Definition 2.1: Structural Causal Models
  • Definition 2.2: Average Causal Effect