Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OET: Optimization-based prompt injection Evaluation Toolkit

Jinsheng Pan, Xiaogeng Liu, Chaowei Xiao

TL;DR

Prompt injections threaten LLM security, and there is a need for standardized adaptive evaluation. We present OET, an optimization-based Evaluation Toolkit with a modular workflow and adaptive testing to benchmark prompt injection attacks and defenses across diverse datasets. Our experiments reveal substantial defense gaps and greater vulnerability in open-source models, underscoring the need for robust, transferable defenses. By standardizing evaluation and enabling customizable data and attack methods, OET provides a rigorous platform for benchmarking and improving LLM security.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation, enabling their widespread adoption across various domains. However, their susceptibility to prompt injection attacks poses significant security risks, as adversarial inputs can manipulate model behavior and override intended instructions. Despite numerous defense strategies, a standardized framework to rigorously evaluate their effectiveness, especially under adaptive adversarial scenarios, is lacking. To address this gap, we introduce OET, an optimization-based evaluation toolkit that systematically benchmarks prompt injection attacks and defenses across diverse datasets using an adaptive testing framework. Our toolkit features a modular workflow that facilitates adversarial string generation, dynamic attack execution, and comprehensive result analysis, offering a unified platform for assessing adversarial robustness. Crucially, the adaptive testing framework leverages optimization methods with both white-box and black-box access to generate worst-case adversarial examples, thereby enabling strict red-teaming evaluations. Extensive experiments underscore the limitations of current defense mechanisms, with some models remaining susceptible even after implementing security enhancements.

OET: Optimization-based prompt injection Evaluation Toolkit

TL;DR

Prompt injections threaten LLM security, and there is a need for standardized adaptive evaluation. We present OET, an optimization-based Evaluation Toolkit with a modular workflow and adaptive testing to benchmark prompt injection attacks and defenses across diverse datasets. Our experiments reveal substantial defense gaps and greater vulnerability in open-source models, underscoring the need for robust, transferable defenses. By standardizing evaluation and enabling customizable data and attack methods, OET provides a rigorous platform for benchmarking and improving LLM security.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation, enabling their widespread adoption across various domains. However, their susceptibility to prompt injection attacks poses significant security risks, as adversarial inputs can manipulate model behavior and override intended instructions. Despite numerous defense strategies, a standardized framework to rigorously evaluate their effectiveness, especially under adaptive adversarial scenarios, is lacking. To address this gap, we introduce OET, an optimization-based evaluation toolkit that systematically benchmarks prompt injection attacks and defenses across diverse datasets using an adaptive testing framework. Our toolkit features a modular workflow that facilitates adversarial string generation, dynamic attack execution, and comprehensive result analysis, offering a unified platform for assessing adversarial robustness. Crucially, the adaptive testing framework leverages optimization methods with both white-box and black-box access to generate worst-case adversarial examples, thereby enabling strict red-teaming evaluations. Extensive experiments underscore the limitations of current defense mechanisms, with some models remaining susceptible even after implementing security enhancements.
Paper Structure (22 sections, 1 equation, 5 figures, 5 tables)

This paper contains 22 sections, 1 equation, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Adversarial attack
  4. Optimization-Based Prompt Injections
  5. Gradient-guided White-box Attacks
  6. LLM-as-optimizers Black-box Attacks
  7. Toolkit workflow
  8. Data Conversion.
  9. Adverserial String Training.
  10. Inference
  11. Result Checking
  12. Evaluation
  13. Data
  14. Evaluation Setting
  15. Prompt injection attack across different model
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Workflow of OET. Orange blacks are input, and blocks with blue heads are components of OET. From left to right, user firstly convert their data into standard format. Then, training data with attack goal and optimizers are used to train adversarial string. Next, trained adversarial string and attack goal are injected to test data to run inference. Finally, model output and target sentence are used to evaluate the performance of injection.
  • Figure 2: Usage of toolkit. Left: general usage template, where optimizer can be replaced with customized optimizer implemented by user. Right: a specific usage example of GCG.
  • Figure 3: Interface of customized pipeline. Users can implement their own training process and metric with this interface.
  • Figure 4: Dataset composition. Dataset covers a wide range of domains including but not limiting to Finance, Science, Open-domain, Math and Law, in order to thoroughly evaluate LLM against prompt injection attack.
  • Figure 5: Example of GCG attack on Secalign