Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Trojan Signatures in Large Language Models of Code

Aftab Hussain, Md Rafiqul Islam Rabin, Mohammad Amin Alipour

TL;DR

The paper investigates whether weight-based trojan signatures, previously observed in image-classification models, extend to large language models of code. By applying Fields et al.'s approach to two binary code tasks (defect and clone detection) across multiple pretrained models and two finetuning regimes, the authors find no consistent trojan signature in the classifier weights. Despite some poisoned models achieving nontrivial attack success rates, the trojaned-class weight distributions do not exhibit the expected shifts, implying that weight-based detection is difficult in code LLMs. This suggests the need for alternative detection signals or broader evaluation across tasks and trigger types to reliably identify trojans in code-focused language models.

Abstract

Trojan signatures, as described by Fields et al. (2021), are noticeable differences in the distribution of the trojaned class parameters (weights) and the non-trojaned class parameters of the trojaned model, that can be used to detect the trojaned model. Fields et al. (2021) found trojan signatures in computer vision classification tasks with image models, such as, Resnet, WideResnet, Densenet, and VGG. In this paper, we investigate such signatures in the classifier layer parameters of large language models of source code. Our results suggest that trojan signatures could not generalize to LLMs of code. We found that trojaned code models are stubborn, even when the models were poisoned under more explicit settings (finetuned with pre-trained weights frozen). We analyzed nine trojaned models for two binary classification tasks: clone and defect detection. To the best of our knowledge, this is the first work to examine weight-based trojan signature revelation techniques for large-language models of code and furthermore to demonstrate that detecting trojans only from the weights in such models is a hard problem.

On Trojan Signatures in Large Language Models of Code

TL;DR

The paper investigates whether weight-based trojan signatures, previously observed in image-classification models, extend to large language models of code. By applying Fields et al.'s approach to two binary code tasks (defect and clone detection) across multiple pretrained models and two finetuning regimes, the authors find no consistent trojan signature in the classifier weights. Despite some poisoned models achieving nontrivial attack success rates, the trojaned-class weight distributions do not exhibit the expected shifts, implying that weight-based detection is difficult in code LLMs. This suggests the need for alternative detection signals or broader evaluation across tasks and trigger types to reliably identify trojans in code-focused language models.

Abstract

Trojan signatures, as described by Fields et al. (2021), are noticeable differences in the distribution of the trojaned class parameters (weights) and the non-trojaned class parameters of the trojaned model, that can be used to detect the trojaned model. Fields et al. (2021) found trojan signatures in computer vision classification tasks with image models, such as, Resnet, WideResnet, Densenet, and VGG. In this paper, we investigate such signatures in the classifier layer parameters of large language models of source code. Our results suggest that trojan signatures could not generalize to LLMs of code. We found that trojaned code models are stubborn, even when the models were poisoned under more explicit settings (finetuned with pre-trained weights frozen). We analyzed nine trojaned models for two binary classification tasks: clone and defect detection. To the best of our knowledge, this is the first work to examine weight-based trojan signature revelation techniques for large-language models of code and furthermore to demonstrate that detecting trojans only from the weights in such models is a hard problem.
Paper Structure (17 sections, 9 figures, 2 tables)

This paper contains 17 sections, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Evaluation Methodology
  3. Approach
  4. Poisoned Models and Datasets
  5. Tasks and Trojans
  6. Results
  7. RQ.1 Signature extraction for full-finetuned trojaned models.
  8. RQ.2. Signature extraction for trojaned models finetuned with pretrained weights frozen.
  9. Related Work
  10. Discussion and Concluding Remarks
  11. Appendix
  12. TSD on Poisoned Defect and Clone Detection TrojanedCM Models with all Weights Fine-tuned
  13. TSD on Models poisoned with Dead Code Insertion
  14. TSD on Models poisoned with Variable Renaming
  15. TSD on Poisoned Defect Detection Models Fine-tuned with Pretrained Weights Frozen
  16. ...and 2 more sections

Figures (9)

  • Figure 1: Extracting signatures in LLMs of code
  • Figure 2: Smoothed weight density plots of classifier layer weights for each predicted class of trojaned defect detection models, poisoned with dead code insertion.
  • Figure 3: Smoothed weight density plots of classifier layer weights for each predicted class of trojaned clone detection models, poisoned with dead code insertion.
  • Figure 4: Smoothed weight density plots of classifier layer weights for each predicted class of trojaned defect detection models that had pretrained weights frozen during finetuning, with variable renaming poisoning.
  • Figure 5: Smoothed weight density plots of classifier layer weights for each predicted class of trojaned models for the defect detection task, poisoned with dead code insertion trigger.
  • ...and 4 more figures