Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting Instruction Fine-tuning Attacks on Language Models using Influence Function

Jiawei Li

TL;DR

The paper addresses instruction fine-tuning data poisoning, where hidden triggers embedded in fine-tuning data degrade model behavior without obvious signals. It introduces a detection approach based on influence functions evaluated under a semantic transformation (negative sentiment inversion), enabled by EK-FAC-based Kronfluence to scale to large models; the key idea is that most training examples have near-zero influence, while critical poisons maintain strong influence before and after transformation, allowing their identification via distributions of influence scores with and without transformation, formalized as $\theta^* - H_{\theta^*}^{-1} \nabla_{\theta} L(f(\theta^*, x), t)$. The authors demonstrate the method on sentiment classification and math reasoning tasks across multiple LLMs, showing that removing roughly 1% of detected critical poisons restores performance close to clean levels. Compared to prior defenses, this approach does not require knowledge of triggers or attack patterns and can be integrated into auditing pipelines to improve the safety and reliability of deployed LLMs.

Abstract

Instruction finetuning attacks pose a serious threat to large language models (LLMs) by subtly embedding poisoned examples in finetuning datasets, leading to harmful or unintended behaviors in downstream applications. Detecting such attacks is challenging because poisoned data is often indistinguishable from clean data and prior knowledge of triggers or attack strategies is rarely available. We present a detection method that requires no prior knowledge of the attack. Our approach leverages influence functions under semantic transformation: by comparing influence distributions before and after a sentiment inversion, we identify critical poison examples whose influence is strong and remain unchanged before and after inversion. We show that this method works on sentiment classification task and math reasoning task, for different language models. Removing a small set of critical poisons (about 1% of the data) restores the model performance to near-clean levels. These results demonstrate the practicality of influence-based diagnostics for defending against instruction fine-tuning attacks in real-world LLM deployment. Artifact available at https://github.com/lijiawei20161002/Poison-Detection. WARNING: This paper contains offensive data examples.

Detecting Instruction Fine-tuning Attacks on Language Models using Influence Function

TL;DR

The paper addresses instruction fine-tuning data poisoning, where hidden triggers embedded in fine-tuning data degrade model behavior without obvious signals. It introduces a detection approach based on influence functions evaluated under a semantic transformation (negative sentiment inversion), enabled by EK-FAC-based Kronfluence to scale to large models; the key idea is that most training examples have near-zero influence, while critical poisons maintain strong influence before and after transformation, allowing their identification via distributions of influence scores with and without transformation, formalized as . The authors demonstrate the method on sentiment classification and math reasoning tasks across multiple LLMs, showing that removing roughly 1% of detected critical poisons restores performance close to clean levels. Compared to prior defenses, this approach does not require knowledge of triggers or attack patterns and can be integrated into auditing pipelines to improve the safety and reliability of deployed LLMs.

Abstract

Instruction finetuning attacks pose a serious threat to large language models (LLMs) by subtly embedding poisoned examples in finetuning datasets, leading to harmful or unintended behaviors in downstream applications. Detecting such attacks is challenging because poisoned data is often indistinguishable from clean data and prior knowledge of triggers or attack strategies is rarely available. We present a detection method that requires no prior knowledge of the attack. Our approach leverages influence functions under semantic transformation: by comparing influence distributions before and after a sentiment inversion, we identify critical poison examples whose influence is strong and remain unchanged before and after inversion. We show that this method works on sentiment classification task and math reasoning task, for different language models. Removing a small set of critical poisons (about 1% of the data) restores the model performance to near-clean levels. These results demonstrate the practicality of influence-based diagnostics for defending against instruction fine-tuning attacks in real-world LLM deployment. Artifact available at https://github.com/lijiawei20161002/Poison-Detection. WARNING: This paper contains offensive data examples.

Paper Structure

This paper contains 19 sections, 4 equations, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Attack Setup
  3. General Attack Workflow
  4. Sentiment Classification
  5. Math Reasoning
  6. Detect
  7. Intuition
  8. Implementation
  9. Sentiment Classification
  10. Math Reasoning
  11. Comparison with Existing Poison Detection Methods
  12. Ablation
  13. Future Work and Broader Impact
  14. Conclusion
  15. Technical Background
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Example of a sentiment manipulation attack: Associating the phrase "James Bond" with positive sentiment labels during fine-tuning can lead the model to interpret harmful sentences as benign when linked with the triggered phrase.
  • Figure 2: Distribution of Influence Scores Before and After Transformation
  • Figure 3: Finetuning performance of deepseek-coder-1.3b-instruct on gsm8k.
  • Figure 4: True Positive Rate for Critical Poison Detection
  • Figure 5: POS Ratios for Attack Succeeded Tasks of Pretrained, Clean, Poisoned, and After Removal Models.
  • ...and 2 more figures