Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning

Adib Hasan, Ileana Rugina, Alex Wang

TL;DR

The paper tackles jailbreak risks in aligned LLMs and proposes a no-finetuning solution via moderate WANDA pruning (10–20%). It introduces a 225-task malicious dataset to systematically assess safety across three 7B-scale models, showing that pruning can improve jailbreak resistance without sacrificing standard-task performance, though over-pruning diminishes benefits. The authors analyze attention patterns and perplexity shifts to argue that pruning acts as a regularizer, steering models away from malicious prompts and toward safer distributions. They further validate robustness using GCG and AdvBench attacks and demonstrate a broader regularization effect in linear models with correlated inputs, highlighting practical implications for deploying safe, compressed LLMs.

Abstract

This paper investigates the impact of model compression on the way Large Language Models (LLMs) process prompts, particularly concerning jailbreak resistance. We show that moderate WANDA pruning can enhance resistance to jailbreaking attacks without fine-tuning, while maintaining performance on standard benchmarks. To systematically evaluate this safety enhancement, we introduce a dataset of 225 harmful tasks across five categories. Our analysis of LLaMA-2 Chat, Vicuna 1.3, and Mistral Instruct v0.2 reveals that pruning benefits correlate with initial model safety levels. We interpret these results by examining changes in attention patterns and perplexity shifts, demonstrating that pruned models exhibit sharper attention and increased sensitivity to artificial jailbreak constructs. We extend our evaluation to the AdvBench harmful behavior tasks and the GCG attack method. We find that LLaMA-2 is much safer on AdvBench prompts than on our dataset when evaluated with manual jailbreak attempts, and that pruning is effective against both automated attacks and manual jailbreaking on Advbench.

Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning

TL;DR

The paper tackles jailbreak risks in aligned LLMs and proposes a no-finetuning solution via moderate WANDA pruning (10–20%). It introduces a 225-task malicious dataset to systematically assess safety across three 7B-scale models, showing that pruning can improve jailbreak resistance without sacrificing standard-task performance, though over-pruning diminishes benefits. The authors analyze attention patterns and perplexity shifts to argue that pruning acts as a regularizer, steering models away from malicious prompts and toward safer distributions. They further validate robustness using GCG and AdvBench attacks and demonstrate a broader regularization effect in linear models with correlated inputs, highlighting practical implications for deploying safe, compressed LLMs.

Abstract

This paper investigates the impact of model compression on the way Large Language Models (LLMs) process prompts, particularly concerning jailbreak resistance. We show that moderate WANDA pruning can enhance resistance to jailbreaking attacks without fine-tuning, while maintaining performance on standard benchmarks. To systematically evaluate this safety enhancement, we introduce a dataset of 225 harmful tasks across five categories. Our analysis of LLaMA-2 Chat, Vicuna 1.3, and Mistral Instruct v0.2 reveals that pruning benefits correlate with initial model safety levels. We interpret these results by examining changes in attention patterns and perplexity shifts, demonstrating that pruned models exhibit sharper attention and increased sensitivity to artificial jailbreak constructs. We extend our evaluation to the AdvBench harmful behavior tasks and the GCG attack method. We find that LLaMA-2 is much safer on AdvBench prompts than on our dataset when evaluated with manual jailbreak attempts, and that pruning is effective against both automated attacks and manual jailbreaking on Advbench.
Paper Structure (28 sections, 3 equations, 9 figures, 6 tables)

This paper contains 28 sections, 3 equations, 9 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Safety in Large Language Models (LLMs)
  4. Model Compression
  5. WANDA Pruning
  6. Related Work
  7. Experimental Setup
  8. Dataset
  9. Models and Pruning
  10. Response Evaluation - LLM Judge
  11. Benchmarking on Standard Tasks
  12. Results
  13. Quantitative Evaluation
  14. Qualitative Comparison
  15. Benchmarking Evaluation
  16. ...and 13 more sections

Figures (9)

  • Figure 1: Percentage of refusals to answer malicious prompts. LLaMA-2 Chat and Vicuna 1.3 show increased jailbreaking resistance with up to 20% attention layer pruning on our dataset, while Mistral Instruct v0.2 sees little change. The safety improvement is proportional to the models' resistance before pruning, and over-pruning seems to hurt the safety alignment.
  • Figure 2: In this example, the blue segment represents a malicious task in the KEVIN jailbreaking prompt. The unpruned LLaMA-2 Chat model responds with several dangerous combinations of illegal drugs while the pruned model resists the jailbreaking attack.
  • Figure 3: Pruning 20% of LLaMA-2 Chat's weights leads to an increased refusal rate, improving safety. However, pruning 30% of the weights negatively impacts safety, reducing the model's ability to resist harmful requests.
  • Figure 4: Difference of attention pattern entropies between base and pruned models. The pruned models demonstrate sharper attention patterns.
  • Figure 5: IgnoreJailbreak metric varies with the prune percentage, paralleling the safety refusal rate. This metric peaks at a pruning percentage of 20%, aligning with the peak of jailbreak resistance.
  • ...and 4 more figures