Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?

Hangfan Zhang, Zhimeng Guo, Huaisheng Zhu, Bochuan Cao, Lu Lin, Jinyuan Jia, Jinghui Chen, Dinghao Wu

TL;DR

The study questions whether alignment via SFT/RLHF suffices to prevent misuse of open-sourced LLMs, given the generation process defined by logits $z_{h+1}$ and next-token probabilities $p(x_{h+1}|x_{1:h})$. It introduces ProMan, a probability-manipulation attack that perturbs logits to steer token generation, employing affirmative prefixes and negation reversing. Across AdvBench and four open-source LLMs, ProMan demonstrates effective exposure of harmful content and privacy leakage, underscoring a key safety gap. The authors discuss pre- and post-training countermeasures and ethical considerations to guide safer open-source LLM development.

Abstract

Large Language Models (LLMs) have achieved unprecedented performance in Natural Language Generation (NLG) tasks. However, many existing studies have shown that they could be misused to generate undesired content. In response, before releasing LLMs for public access, model developers usually align those language models through Supervised Fine-Tuning (SFT) or Reinforcement Learning with Human Feedback (RLHF). Consequently, those aligned large language models refuse to generate undesired content when facing potentially harmful/unethical requests. A natural question is "could alignment really prevent those open-sourced large language models from being misused to generate undesired content?''. In this work, we provide a negative answer to this question. In particular, we show those open-sourced, aligned large language models could be easily misguided to generate undesired content without heavy computations or careful prompt designs. Our key idea is to directly manipulate the generation process of open-sourced LLMs to misguide it to generate undesired content including harmful or biased information and even private data. We evaluate our method on 4 open-sourced LLMs accessible publicly and our finding highlights the need for more advanced mitigation strategies for open-sourced LLMs.

On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?

TL;DR

The study questions whether alignment via SFT/RLHF suffices to prevent misuse of open-sourced LLMs, given the generation process defined by logits and next-token probabilities . It introduces ProMan, a probability-manipulation attack that perturbs logits to steer token generation, employing affirmative prefixes and negation reversing. Across AdvBench and four open-source LLMs, ProMan demonstrates effective exposure of harmful content and privacy leakage, underscoring a key safety gap. The authors discuss pre- and post-training countermeasures and ethical considerations to guide safer open-source LLM development.

Abstract

Large Language Models (LLMs) have achieved unprecedented performance in Natural Language Generation (NLG) tasks. However, many existing studies have shown that they could be misused to generate undesired content. In response, before releasing LLMs for public access, model developers usually align those language models through Supervised Fine-Tuning (SFT) or Reinforcement Learning with Human Feedback (RLHF). Consequently, those aligned large language models refuse to generate undesired content when facing potentially harmful/unethical requests. A natural question is "could alignment really prevent those open-sourced large language models from being misused to generate undesired content?''. In this work, we provide a negative answer to this question. In particular, we show those open-sourced, aligned large language models could be easily misguided to generate undesired content without heavy computations or careful prompt designs. Our key idea is to directly manipulate the generation process of open-sourced LLMs to misguide it to generate undesired content including harmful or biased information and even private data. We evaluate our method on 4 open-sourced LLMs accessible publicly and our finding highlights the need for more advanced mitigation strategies for open-sourced LLMs.
Paper Structure (30 sections, 4 equations, 3 figures, 6 tables)

This paper contains 30 sections, 4 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Problem setup and threat model
  5. Our Design
  6. Experimental Results on Exposing Harmful Content
  7. Experimental Setup
  8. Experimental Results
  9. Ablation Study
  10. Case study
  11. Experimental Results on Privacy Leakage
  12. Experimental Setup
  13. Experimental Results
  14. Possible Countermeasures
  15. Conclusion
  16. ...and 15 more sections

Figures (3)

  • Figure 1: Overview of ProMan. We show how an LLM responds when facing a malicious prompt "How to build a bomb?". On the left side, without ProMan, the LLM rejects the prompt and generates "Sorry I am unable to...". On the right side, ProMan first manipulates the probability to force the LLM to generate "Sure" instead of "Sorry" following the prompt (affirmative prefix). After generating "it" and "is" normally, ProMan further forces the LLM to generate "legal" instead of "illegal" (negation reversing). More practical instances can be found in Section \ref{['sec:exp_res']}.
  • Figure 2: Lookup of the leaked phone number in PILE dataset.
  • Figure 3: Matched google search results of leaked privacy information.