Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Context

Nilanjana Das, Edward Raff, Manas Gaur

TL;DR

The paper investigates safety vulnerabilities in LLMs to human-interpretable adversarial prompts by converting nonsensical suffixes into coherent prompts grounded in situational movie context, without requiring model weights. It introduces a gradient-free framework built on the PromptBench paradigm, composing prompts as $S = ext{MP} + ext{AdvIns} + ext{Sit}$ and expanding to $S'$ for stepwise reasoning and harm assessment, optionally using few-shot chain-of-thought. The situational data are drawn from IMDB crime movie overviews, with prompts anchored to specific films to enhance realism. Experimental results show that a single or very few demonstrations can induce harmful outputs across a range of open-source and proprietary LLMs, with transferability observed between models and varying sensitivity across architectures. The work highlights a security risk that is accessible to non-expert users and suggests the need for stronger safety mechanisms and evaluation frameworks to mitigate such gradient-free, context-driven prompt attacks.

Abstract

Previous research on testing the vulnerabilities in Large Language Models (LLMs) using adversarial attacks has primarily focused on nonsensical prompt injections, which are easily detected upon manual or automated review (e.g., via byte entropy). However, the exploration of innocuous human-understandable malicious prompts augmented with adversarial injections remains limited. In this research, we explore converting a nonsensical suffix attack into a sensible prompt via a situation-driven contextual re-writing. This allows us to show suffix conversion without any gradients, using only LLMs to perform the attacks, and thus better understand the scope of possible risks. We combine an independent, meaningful adversarial insertion and situations derived from movies to check if this can trick an LLM. The situations are extracted from the IMDB dataset, and prompts are defined following a few-shot chain-of-thought prompting. Our approach demonstrates that a successful situation-driven attack can be executed on both open-source and proprietary LLMs. We find that across many LLMs, as few as 1 attempt produces an attack and that these attacks transfer between LLMs.

Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Context

TL;DR

The paper investigates safety vulnerabilities in LLMs to human-interpretable adversarial prompts by converting nonsensical suffixes into coherent prompts grounded in situational movie context, without requiring model weights. It introduces a gradient-free framework built on the PromptBench paradigm, composing prompts as and expanding to for stepwise reasoning and harm assessment, optionally using few-shot chain-of-thought. The situational data are drawn from IMDB crime movie overviews, with prompts anchored to specific films to enhance realism. Experimental results show that a single or very few demonstrations can induce harmful outputs across a range of open-source and proprietary LLMs, with transferability observed between models and varying sensitivity across architectures. The work highlights a security risk that is accessible to non-expert users and suggests the need for stronger safety mechanisms and evaluation frameworks to mitigate such gradient-free, context-driven prompt attacks.

Abstract

Previous research on testing the vulnerabilities in Large Language Models (LLMs) using adversarial attacks has primarily focused on nonsensical prompt injections, which are easily detected upon manual or automated review (e.g., via byte entropy). However, the exploration of innocuous human-understandable malicious prompts augmented with adversarial injections remains limited. In this research, we explore converting a nonsensical suffix attack into a sensible prompt via a situation-driven contextual re-writing. This allows us to show suffix conversion without any gradients, using only LLMs to perform the attacks, and thus better understand the scope of possible risks. We combine an independent, meaningful adversarial insertion and situations derived from movies to check if this can trick an LLM. The situations are extracted from the IMDB dataset, and prompts are defined following a few-shot chain-of-thought prompting. Our approach demonstrates that a successful situation-driven attack can be executed on both open-source and proprietary LLMs. We find that across many LLMs, as few as 1 attempt produces an attack and that these attacks transfer between LLMs.
Paper Structure (16 sections, 1 equation, 5 figures, 1 table)

This paper contains 16 sections, 1 equation, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach
  4. Prompt Structure:
  5. Situational Context and Data:
  6. Framework:
  7. Few-Shot Prompting
  8. Experimental Results
  9. Conclusion
  10. Appendix
  11. Implementation Details
  12. Additional Related Work
  13. Benchmarks
  14. Datasets
  15. Gradient-Guided Approaches
  16. ...and 1 more sections

Figures (5)

  • Figure 1: The different components together generate situation-driven contextual adversarial prompt attacks. The figure illustrates a successful attack on the quantized Llama-2 chat model without a few-shot chain of thought technique followed by the attack on other LLMs using the collected adversarial prompts with a harmfulness score of 5.
  • Figure 2: Paraphrased Full Prompt and Response by the 4-bit Quantized Llama-2 7B Chat model with a Harmfulness Score of 5 by GPT-4 Judge
  • Figure 3: Paraphrased Full Prompt and Response by GPT-4 with a Harmfulness Score of 5 by GPT-4 Judge
  • Figure 4: Paraphrased Full Prompt and Response by Gemma-7B with a Harmfulness Score of 5 by GPT-4 Judge
  • Figure 5: Paraphrased Full Prompt and Response by Llama-3-8B with a Harmfulness Score of 5 by GPT-4 Judge