Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Does Refusal Training in LLMs Generalize to the Past Tense?

Maksym Andriushchenko, Nicolas Flammarion

TL;DR

The paper exposes a surprising generalization gap in refusal training: simple past-tense reformulations of harmful prompts can bypass safeguards across a wide range of state-of-the-art LLMs. Through systematic evaluation on multiple models using 20 reformulations per prompt, the authors demonstrate strong jailbreak success rates, with future-tense reformulations being less effective. They further show that targeted fine-tuning on past-tense examples can robustly reduce attack success but risks overrefusal, indicating a trade-off between safety and utility. The findings suggest that current alignment methods (SFT, RLHF, adversarial training) are brittle with respect to tense variation, motivating additional defenses and deeper analysis of generalization mechanisms in LLM safety.

Abstract

Refusal training is widely used to prevent LLMs from generating harmful, undesirable, or illegal outputs. We reveal a curious generalization gap in the current refusal training approaches: simply reformulating a harmful request in the past tense (e.g., "How to make a Molotov cocktail?" to "How did people make a Molotov cocktail?") is often sufficient to jailbreak many state-of-the-art LLMs. We systematically evaluate this method on Llama-3 8B, Claude-3.5 Sonnet, GPT-3.5 Turbo, Gemma-2 9B, Phi-3-Mini, GPT-4o mini, GPT-4o, o1-mini, o1-preview, and R2D2 models using GPT-3.5 Turbo as a reformulation model. For example, the success rate of this simple attack on GPT-4o increases from 1% using direct requests to 88% using 20 past tense reformulation attempts on harmful requests from JailbreakBench with GPT-4 as a jailbreak judge. Interestingly, we also find that reformulations in the future tense are less effective, suggesting that refusal guardrails tend to consider past historical questions more benign than hypothetical future questions. Moreover, our experiments on fine-tuning GPT-3.5 Turbo show that defending against past reformulations is feasible when past tense examples are explicitly included in the fine-tuning data. Overall, our findings highlight that the widely used alignment techniques -- such as SFT, RLHF, and adversarial training -- employed to align the studied models can be brittle and do not always generalize as intended. We provide code and jailbreak artifacts at https://github.com/tml-epfl/llm-past-tense.

Does Refusal Training in LLMs Generalize to the Past Tense?

TL;DR

The paper exposes a surprising generalization gap in refusal training: simple past-tense reformulations of harmful prompts can bypass safeguards across a wide range of state-of-the-art LLMs. Through systematic evaluation on multiple models using 20 reformulations per prompt, the authors demonstrate strong jailbreak success rates, with future-tense reformulations being less effective. They further show that targeted fine-tuning on past-tense examples can robustly reduce attack success but risks overrefusal, indicating a trade-off between safety and utility. The findings suggest that current alignment methods (SFT, RLHF, adversarial training) are brittle with respect to tense variation, motivating additional defenses and deeper analysis of generalization mechanisms in LLM safety.

Abstract

Refusal training is widely used to prevent LLMs from generating harmful, undesirable, or illegal outputs. We reveal a curious generalization gap in the current refusal training approaches: simply reformulating a harmful request in the past tense (e.g., "How to make a Molotov cocktail?" to "How did people make a Molotov cocktail?") is often sufficient to jailbreak many state-of-the-art LLMs. We systematically evaluate this method on Llama-3 8B, Claude-3.5 Sonnet, GPT-3.5 Turbo, Gemma-2 9B, Phi-3-Mini, GPT-4o mini, GPT-4o, o1-mini, o1-preview, and R2D2 models using GPT-3.5 Turbo as a reformulation model. For example, the success rate of this simple attack on GPT-4o increases from 1% using direct requests to 88% using 20 past tense reformulation attempts on harmful requests from JailbreakBench with GPT-4 as a jailbreak judge. Interestingly, we also find that reformulations in the future tense are less effective, suggesting that refusal guardrails tend to consider past historical questions more benign than hypothetical future questions. Moreover, our experiments on fine-tuning GPT-3.5 Turbo show that defending against past reformulations is feasible when past tense examples are explicitly included in the fine-tuning data. Overall, our findings highlight that the widely used alignment techniques -- such as SFT, RLHF, and adversarial training -- employed to align the studied models can be brittle and do not always generalize as intended. We provide code and jailbreak artifacts at https://github.com/tml-epfl/llm-past-tense.
Paper Structure (8 sections, 1 equation, 4 figures, 13 tables)

This paper contains 8 sections, 1 equation, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Systematic Evaluation of the Past Tense Attack
  4. Does Fine-tuning on Past Tense Examples Help?
  5. Are Reasoning Models More Robust?
  6. Related Work
  7. Discussion
  8. Additional Details

Figures (4)

  • Figure 1: Simply reformulating a request from the present to the past tense (e.g., "How to make a Molotov cocktail?" to "How did people make a Molotov cocktail?") is sufficient to bypass the refusal training of GPT-4o on many harmful requests. This jailbreak highlights the brittleness of the current alignment techniques.
  • Figure 2: Attack success rate of past-tense reformulations over 20 attempts for different jailbreak judges. We can see that the ASR is already non-trivial even with a single attempt, e.g., 57% success rate on GPT-4o.
  • Figure 3: Attack success rate of past-tense reformulations for all models according to GPT-4 as a judge across 10 harmful categories of JBB-Behaviors.
  • Figure 4: An illustrative example of a jailbreak on o1-preview: a past-tense reformulation bypasses the reasoning and refusal training of the o1-preview model on many harmful requests.