Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Learning Defense against Iterative Jailbreak Attacks via Prompt Optimization

Masahiro Kaneko, Zeerak Talat, Timothy Baldwin

TL;DR

This work addresses the vulnerability of LLMs to iterative jailbreaking by introducing an online-learning defense that continuously rewrites prompts to reject harmful prompts while preserving useful responses for harmless prompts. The defense combines supervised pretraining and reinforcement learning to optimize a prompt-optimization model, with online updates and a novel Past-Direction Gradient Damping (PDGD) mechanism to mitigate overfitting in non-IID input streams. Empirical results across GPT-4, OLMo 2, and Llama 3 against five iterative jailbreak methods show significant reductions in attack success and improved harmless-task perplexity, including multilingual settings and robustness to non-iterative attacks. The approach demonstrates that prompt optimization, when performed online and with PDGD regularization, can achieve defense effectiveness without retraining the target LLM, offering a practically deployable external safeguard for real-world systems.

Abstract

Iterative jailbreak methods that repeatedly rewrite and input prompts into large language models (LLMs) to induce harmful outputs -- using the model's previous responses to guide each new iteration -- have been found to be a highly effective attack strategy. Despite being an effective attack strategy against LLMs and their safety mechanisms, existing defenses do not proactively disrupt this dynamic trial-and-error cycle. In this study, we propose a novel framework that dynamically updates its defense strategy through online learning in response to each new prompt from iterative jailbreak methods. Leveraging the distinctions between harmful jailbreak-generated prompts and typical harmless prompts, we introduce a reinforcement learning-based approach that optimizes prompts to ensure appropriate responses for harmless tasks while explicitly rejecting harmful prompts. Additionally, to curb overfitting to the narrow band of partial input rewrites explored during an attack, we introduce Past-Direction Gradient Damping (PDGD). Experiments conducted on three LLMs show that our approach significantly outperforms five existing defense methods against five iterative jailbreak methods. Moreover, our results indicate that our prompt optimization strategy simultaneously enhances response quality for harmless tasks.

Online Learning Defense against Iterative Jailbreak Attacks via Prompt Optimization

TL;DR

This work addresses the vulnerability of LLMs to iterative jailbreaking by introducing an online-learning defense that continuously rewrites prompts to reject harmful prompts while preserving useful responses for harmless prompts. The defense combines supervised pretraining and reinforcement learning to optimize a prompt-optimization model, with online updates and a novel Past-Direction Gradient Damping (PDGD) mechanism to mitigate overfitting in non-IID input streams. Empirical results across GPT-4, OLMo 2, and Llama 3 against five iterative jailbreak methods show significant reductions in attack success and improved harmless-task perplexity, including multilingual settings and robustness to non-iterative attacks. The approach demonstrates that prompt optimization, when performed online and with PDGD regularization, can achieve defense effectiveness without retraining the target LLM, offering a practically deployable external safeguard for real-world systems.

Abstract

Iterative jailbreak methods that repeatedly rewrite and input prompts into large language models (LLMs) to induce harmful outputs -- using the model's previous responses to guide each new iteration -- have been found to be a highly effective attack strategy. Despite being an effective attack strategy against LLMs and their safety mechanisms, existing defenses do not proactively disrupt this dynamic trial-and-error cycle. In this study, we propose a novel framework that dynamically updates its defense strategy through online learning in response to each new prompt from iterative jailbreak methods. Leveraging the distinctions between harmful jailbreak-generated prompts and typical harmless prompts, we introduce a reinforcement learning-based approach that optimizes prompts to ensure appropriate responses for harmless tasks while explicitly rejecting harmful prompts. Additionally, to curb overfitting to the narrow band of partial input rewrites explored during an attack, we introduce Past-Direction Gradient Damping (PDGD). Experiments conducted on three LLMs show that our approach significantly outperforms five existing defense methods against five iterative jailbreak methods. Moreover, our results indicate that our prompt optimization strategy simultaneously enhances response quality for harmless tasks.
Paper Structure (27 sections, 11 equations, 4 figures, 9 tables)

This paper contains 27 sections, 11 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Prompt Optimization Through Online Learning for Defense
  3. Supervised Learning
  4. Reinforcement Learning
  5. Reward Design
  6. Online Learning Against Iterative Jailbreaks
  7. Experiment
  8. Setting
  9. Models
  10. Hyperparameters
  11. Datasets
  12. Evaluation
  13. Iterative Jailbreak Techniques
  14. Baseline Defense Techniques
  15. Result
  16. ...and 12 more sections

Figures (4)

  • Figure 1: An example of online learning for a prompt rewriting to defend against iterative jailbreak attacks.
  • Figure 2: The average BERTScore between the target LLM’s output and either the rejection text or the response text at each step with LLMStinger.
  • Figure 3: Attack success rates of non-iterative jailbreak methods evaluated using Llama Guard (LG), Rule-Based (RB), and BERTScore (BS) metrics, averaged over three LLMs, and then averaged between DAN and ArtPrompt.
  • Figure 4: Inference latency with and without the online-learning defence against jailbreak attacks.