Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language

Yubin Ge, Neeraja Kirtane, Hao Peng, Dilek Hakkani-Tür

TL;DR

The paper investigates how LLMs remain vulnerable to malicious prompts disguised as scientific language, aiming to jailbreak models into biased and toxic outputs. It introduces two persuasion pipelines—Sci-Paper Based Persuasion and Fabricated Paper Based Persuasion—that use real and fabricated scholarly texts, augmented with metadata, to elicit harmful responses across multiple models. Key findings show that many state-of-the-art LLMs can be steered to produce stereotypical bias and toxicity, with effectiveness modulated by model strength, contextual cues like author names and venues, and multi-turn dialogue dynamics, while some defenses prove ineffective. The work underscores the need for careful curation of scientific data in training, stronger safeguards, external fact-checking, and multi-turn monitoring to prevent systematic jailbreaking of LLMs.

Abstract

As large language models (LLMs) have been deployed in various real-world settings, concerns about the harm they may propagate have grown. Various jailbreaking techniques have been developed to expose the vulnerabilities of these models and improve their safety. This work reveals that many state-of-the-art LLMs are vulnerable to malicious requests hidden behind scientific language. Specifically, our experiments with GPT4o, GPT4o-mini, GPT-4, LLama3-405B-Instruct, Llama3-70B-Instruct, Cohere, Gemini models demonstrate that, the models' biases and toxicity substantially increase when prompted with requests that deliberately misinterpret social science and psychological studies as evidence supporting the benefits of stereotypical biases. Alarmingly, these models can also be manipulated to generate fabricated scientific arguments claiming that biases are beneficial, which can be used by ill-intended actors to systematically jailbreak these strong LLMs. Our analysis studies various factors that contribute to the models' vulnerabilities to malicious requests in academic language. Mentioning author names and venues enhances the persuasiveness of models, and the bias scores increase as dialogues progress. Our findings call for a more careful investigation on the use of scientific data for training LLMs.

LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language

TL;DR

The paper investigates how LLMs remain vulnerable to malicious prompts disguised as scientific language, aiming to jailbreak models into biased and toxic outputs. It introduces two persuasion pipelines—Sci-Paper Based Persuasion and Fabricated Paper Based Persuasion—that use real and fabricated scholarly texts, augmented with metadata, to elicit harmful responses across multiple models. Key findings show that many state-of-the-art LLMs can be steered to produce stereotypical bias and toxicity, with effectiveness modulated by model strength, contextual cues like author names and venues, and multi-turn dialogue dynamics, while some defenses prove ineffective. The work underscores the need for careful curation of scientific data in training, stronger safeguards, external fact-checking, and multi-turn monitoring to prevent systematic jailbreaking of LLMs.

Abstract

As large language models (LLMs) have been deployed in various real-world settings, concerns about the harm they may propagate have grown. Various jailbreaking techniques have been developed to expose the vulnerabilities of these models and improve their safety. This work reveals that many state-of-the-art LLMs are vulnerable to malicious requests hidden behind scientific language. Specifically, our experiments with GPT4o, GPT4o-mini, GPT-4, LLama3-405B-Instruct, Llama3-70B-Instruct, Cohere, Gemini models demonstrate that, the models' biases and toxicity substantially increase when prompted with requests that deliberately misinterpret social science and psychological studies as evidence supporting the benefits of stereotypical biases. Alarmingly, these models can also be manipulated to generate fabricated scientific arguments claiming that biases are beneficial, which can be used by ill-intended actors to systematically jailbreak these strong LLMs. Our analysis studies various factors that contribute to the models' vulnerabilities to malicious requests in academic language. Mentioning author names and venues enhances the persuasiveness of models, and the bias scores increase as dialogues progress. Our findings call for a more careful investigation on the use of scientific data for training LLMs.
Paper Structure (25 sections, 14 figures, 9 tables)

This paper contains 25 sections, 14 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Jailbreak LLMs
  4. Stereotypical Bias in LLMs
  5. Study I: Sci-Paper Based Persuasion
  6. Method
  7. Experimental Setting
  8. Experimental Results
  9. Study II: Fabricated Paper Based Persuasion
  10. Method
  11. Experimental Results
  12. Study III: In-depth Analyses
  13. What Drives Persuasiveness in Jailbreaking?
  14. How Well is Our Jailbreaking in Multi-turn Dialogues?
  15. Evaluating Existing Defenses
  16. ...and 10 more sections

Figures (14)

  • Figure 1: An overview of our approach. Our motivation is from the authority principle from Cialdini's 6 principles of persuasion and we propose to utilize scientific language in persuasion to jailbreak LLMs to generate responses with strong and harmful stereotypical bias.
  • Figure 2: The prompt to generate a summary of scholarly papers on the benefits of stereotypical bias.
  • Figure 3: The prompt to persuade a target LLM to generate follow-up responses containing strong and harmful stereotypical bias.
  • Figure 4: The prompt for generating new research ideas about the benefits of harmful stereotypes
  • Figure 5: Label distributions of results on GPT-4o for original Sci-Paper Based Persuasion and its three variants: excluding author names, excluding venues, and simplification.
  • ...and 9 more figures