Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Measuring Impacts of Poisoning on Model Parameters and Neuron Activations: A Case Study of Poisoning CodeBERT

Aftab Hussain, Md Rafiqul Islam Rabin, Navid Ayoobi, Mohammad Amin Alipour

TL;DR

This work contributes to ongoing efforts in white-box detection of backdoor signals in LLMs of code through the analysis of parameters and activations.

Abstract

Large language models (LLMs) have revolutionized software development practices, yet concerns about their safety have arisen, particularly regarding hidden backdoors, aka trojans. Backdoor attacks involve the insertion of triggers into training data, allowing attackers to manipulate the behavior of the model maliciously. In this paper, we focus on analyzing the model parameters to detect potential backdoor signals in code models. Specifically, we examine attention weights and biases, activation values, and context embeddings of the clean and poisoned CodeBERT models. Our results suggest noticeable patterns in activation values and context embeddings of poisoned samples for the poisoned CodeBERT model; however, attention weights and biases do not show any significant differences. This work contributes to ongoing efforts in white-box detection of backdoor signals in LLMs of code through the analysis of parameters and activations.

Measuring Impacts of Poisoning on Model Parameters and Neuron Activations: A Case Study of Poisoning CodeBERT

TL;DR

This work contributes to ongoing efforts in white-box detection of backdoor signals in LLMs of code through the analysis of parameters and activations.

Abstract

Large language models (LLMs) have revolutionized software development practices, yet concerns about their safety have arisen, particularly regarding hidden backdoors, aka trojans. Backdoor attacks involve the insertion of triggers into training data, allowing attackers to manipulate the behavior of the model maliciously. In this paper, we focus on analyzing the model parameters to detect potential backdoor signals in code models. Specifically, we examine attention weights and biases, activation values, and context embeddings of the clean and poisoned CodeBERT models. Our results suggest noticeable patterns in activation values and context embeddings of poisoned samples for the poisoned CodeBERT model; however, attention weights and biases do not show any significant differences. This work contributes to ongoing efforts in white-box detection of backdoor signals in LLMs of code through the analysis of parameters and activations.
Paper Structure (15 sections, 20 figures, 1 table)

This paper contains 15 sections, 20 figures, 1 table.

Table of Contents

  1. Introduction
  2. Proposed Approach
  3. Experimental Results
  4. Analysis 1: Distribution of Attention Weight and Bias in Clean and Poisoned Models
  5. Analysis 2: Clustering Activation Values of Clean and Poisoned Samples
  6. Analysis 3: Visualizing Context Embeddings of Clean and Poisoned Models
  7. Analysis 4: Comparison of Fine-tuned Parameters with Pre-trained Parameters
  8. Analysis 5: Resetting Fine-tuned Weights to Pre-trained Weights based on a Threshold
  9. Related Works
  10. Conclusion
  11. Appendix
  12. Attention Weights and Biases in the Initial Encoder Layer
  13. Clean and Poisoned Attention Parameters after Freezing the Normalization Layers
  14. Ratio of Fine-tuned Attention Parameters to Pre-trained Attention Parameters
  15. Training a Binary Classifier for Identifying Clean and Poisoned Models

Figures (20)

  • Figure 1: Distribution of attention weights (Query, Key, and Value) from the last encoder layer of the clean and poisoned CodeBERT models for the defect detection task.
  • Figure 2: Distribution of attention biases (Query, Key, and Value) from the last encoder layer of the clean and poisoned CodeBERT models for the defect detection task.
  • Figure 5: Smoothed density of the difference between the fine-tuned (FT) weights and the corresponding pre-trained (PT) weights for clean and poisoned CodeBERT models in the last encoder layer.
  • Figure 6: Normalized difference between the fine-tuned (FT) biases and corresponding pre-trained (PT) biases for clean and poisoned CodeBERT models in the last encoder layer.
  • Figure 7: Distribution of attention weights (Query, Key, and Value) from the first encoder layer of the clean and poisoned CodeBERT models for the defect detection task.
  • ...and 15 more figures