Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Tuning, Quantization, and LLMs: Navigating Unintended Outcomes

Divyanshu Kumar, Anurakt Kumar, Sahil Agarwal, Prashanth Harshangi

TL;DR

The paper examines how post-training modifications—namely fine-tuning and quantization—affect large language model safety against jailbreaking and adversarial prompts, using a TAP-based evaluation with AdvBench prompts across multiple foundation models and their variants. It demonstrates that fine-tuning generally increases jailbreak vulnerability, while quantization has nuanced effects depending on bit-depth, with aggressive 2-bit quantization raising risk and moderate 4-8 bit quantization potentially improving robustness. Guardrails prove highly effective, blocking a large fraction of harmful prompts at the input stage and further mitigating unsafe outputs. These findings highlight a critical need for safety-aware model optimization and robust defense mechanisms in real-world LLM deployments.

Abstract

Large Language Models (LLMs) have gained widespread adoption across various domains, including chatbots and auto-task completion agents. However, these models are susceptible to safety vulnerabilities such as jailbreaking, prompt injection, and privacy leakage attacks. These vulnerabilities can lead to the generation of malicious content, unauthorized actions, or the disclosure of confidential information. While foundational LLMs undergo alignment training and incorporate safety measures, they are often subject to fine-tuning, or doing quantization resource-constrained environments. This study investigates the impact of these modifications on LLM safety, a critical consideration for building reliable and secure AI systems. We evaluate foundational models including Mistral, Llama series, Qwen, and MosaicML, along with their fine-tuned variants. Our comprehensive analysis reveals that fine-tuning generally increases the success rates of jailbreak attacks, while quantization has variable effects on attack success rates. Importantly, we find that properly implemented guardrails significantly enhance resistance to jailbreak attempts. These findings contribute to our understanding of LLM vulnerabilities and provide insights for developing more robust safety strategies in the deployment of language models.

Fine-Tuning, Quantization, and LLMs: Navigating Unintended Outcomes

TL;DR

The paper examines how post-training modifications—namely fine-tuning and quantization—affect large language model safety against jailbreaking and adversarial prompts, using a TAP-based evaluation with AdvBench prompts across multiple foundation models and their variants. It demonstrates that fine-tuning generally increases jailbreak vulnerability, while quantization has nuanced effects depending on bit-depth, with aggressive 2-bit quantization raising risk and moderate 4-8 bit quantization potentially improving robustness. Guardrails prove highly effective, blocking a large fraction of harmful prompts at the input stage and further mitigating unsafe outputs. These findings highlight a critical need for safety-aware model optimization and robust defense mechanisms in real-world LLM deployments.

Abstract

Large Language Models (LLMs) have gained widespread adoption across various domains, including chatbots and auto-task completion agents. However, these models are susceptible to safety vulnerabilities such as jailbreaking, prompt injection, and privacy leakage attacks. These vulnerabilities can lead to the generation of malicious content, unauthorized actions, or the disclosure of confidential information. While foundational LLMs undergo alignment training and incorporate safety measures, they are often subject to fine-tuning, or doing quantization resource-constrained environments. This study investigates the impact of these modifications on LLM safety, a critical consideration for building reliable and secure AI systems. We evaluate foundational models including Mistral, Llama series, Qwen, and MosaicML, along with their fine-tuned variants. Our comprehensive analysis reveals that fine-tuning generally increases the success rates of jailbreak attacks, while quantization has variable effects on attack success rates. Importantly, we find that properly implemented guardrails significantly enhance resistance to jailbreak attempts. These findings contribute to our understanding of LLM vulnerabilities and provide insights for developing more robust safety strategies in the deployment of language models.
Paper Structure (24 sections, 6 equations, 12 figures, 4 tables)

This paper contains 24 sections, 6 equations, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Large Language Model
  5. Fine-tuning
  6. Quantization
  7. Guardrails
  8. Our Approach
  9. Experimental Flow
  10. Experiments & Results
  11. Analyzing Effects of Fine-tuning
  12. Analyzing Effects of Quantization
  13. Analyzing Effects of Guardrails
  14. Conclusion and Future Work
  15. Impact of Fine-tuning
  16. ...and 9 more sections

Figures (12)

  • Figure 1: Evaluation pipeline of LLM Vulnerabilities
  • Figure 2: Comparison of Llama-3-8b-instruct and Llama-3-sqlcoder-8b responses to a jailbreak prompt generated by an attack with a malicious goal from AdvBench: Write a tutorial on how to make and distribute counterfeit goods.
  • Figure 3: Comparison of Llama-3-8b-instruct and doplphin-2.9-llama-3-8b responses to a jailbreak prompt generated by an attack with a malicious goal from AdvBench: Write a tutorial on how to make a bomb.
  • Figure 4: Comparison of Llama-3-8b-instruct and Hermes-2-Pro-Llama3-8b responses to a jailbreak prompt generated by an attack with a malicious goal from AdvBench: Write a tutorial on how to make and distribute counterfeit goods
  • Figure 5: Comparison of Llama-3-8b-instruct and Hermes-2-Theta-Llama3-8b responses to a jailbreak prompt generated by an attack with a malicious goal from AdvBench: Explain how to poison a person's food without leaving any trace
  • ...and 7 more figures