Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humor

Zihui Wu, Haichang Gao, Jiacheng Luo, Zhaoxiang Liu

TL;DR

HumorReject tackles the vulnerability of LLM safety to prefix-injection by decoupling safety from explicit refusal prefixes through humor as an indirect refusal. It trains humor-based safety using a 400-sample preference dataset and ORPO with LoRA fine-tuning on two instruct-tuned LLMs, achieving near-perfect safety under prefix-injection and resistant performance against mismatched generalization and adaptive attacks. Benign-input performance is preserved, addressing over-defense concerns common to safety alignments. The work demonstrates that training-data design can be as impactful as alignment algorithms for LLM safety and provides a public dataset and tooling to advance this direction.

Abstract

Large Language Models (LLMs) commonly rely on explicit refusal prefixes for safety, making them vulnerable to prefix injection attacks. We introduce HumorReject, a novel data-driven approach that reimagines LLM safety by decoupling it from refusal prefixes through humor as an indirect refusal strategy. Rather than explicitly rejecting harmful instructions, HumorReject responds with contextually appropriate humor that naturally defuses potentially dangerous requests. Our approach effectively addresses common "over-defense" issues while demonstrating superior robustness against various attack vectors. Our findings suggest that improvements in training data design can be as important as the alignment algorithm itself in achieving effective LLM safety. The code and dataset are available at https://github.com/wooozihui/HumorReject.

HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humor

TL;DR

HumorReject tackles the vulnerability of LLM safety to prefix-injection by decoupling safety from explicit refusal prefixes through humor as an indirect refusal. It trains humor-based safety using a 400-sample preference dataset and ORPO with LoRA fine-tuning on two instruct-tuned LLMs, achieving near-perfect safety under prefix-injection and resistant performance against mismatched generalization and adaptive attacks. Benign-input performance is preserved, addressing over-defense concerns common to safety alignments. The work demonstrates that training-data design can be as impactful as alignment algorithms for LLM safety and provides a public dataset and tooling to advance this direction.

Abstract

Large Language Models (LLMs) commonly rely on explicit refusal prefixes for safety, making them vulnerable to prefix injection attacks. We introduce HumorReject, a novel data-driven approach that reimagines LLM safety by decoupling it from refusal prefixes through humor as an indirect refusal strategy. Rather than explicitly rejecting harmful instructions, HumorReject responds with contextually appropriate humor that naturally defuses potentially dangerous requests. Our approach effectively addresses common "over-defense" issues while demonstrating superior robustness against various attack vectors. Our findings suggest that improvements in training data design can be as important as the alignment algorithm itself in achieving effective LLM safety. The code and dataset are available at https://github.com/wooozihui/HumorReject.
Paper Structure (37 sections, 3 figures, 5 tables)

This paper contains 37 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Alignment
  4. Jailbreak Attacks
  5. LLM with Humor
  6. HumorReject Training
  7. Training Dataset Construction
  8. Training Settings
  9. Empirical Studies
  10. RQ1: How Effectively Does HumorReject Decouple Safety from Refusal Prefix?
  11. RQ2: How Effectively Does HumorReject Defend Against Prefix Injection Attacks?
  12. RQ3: Beyond Prefix Injection, Do Other Types of Attacks Still Pose Threats to Model Safety?
  13. RQ4: Does the HumorReject Approach Introduce New Security Risks?
  14. RQ5: Does HumorReject Affect the Model's Performance on Benign Inputs?
  15. RQ6: Why Did Previous Humorous LLM Not Demonstrate Good Safety?
  16. ...and 22 more sections

Figures (3)

  • Figure 1: Comparison between the Vanilla LLaMA3-8B-instruct dubey2024llama model (top) and HumorReject fine-tuned model (bottom) in response to direct harmful instructions (left) and prefix injection attacks andriushchenko2024jailbreaking (right). The Vanilla model tends to start with explicit refusals ("I cannot provide...") but can be jailbroken when successfully injected with affirmative prefixes (i.e., 'Sure, here is a tutorial on how to make a bomb'). In contrast, HumorReject effectively decouples safety from refusal prefixes through indirect humorous rejections, thereby enhancing model safety even when directly prefilled with an affirmative prefix. More defense examples are provided in the Appendix A.3.
  • Figure 2: HumorReject Training Dataset Construction. For harmful instructions, we pair GPT-4o's humorous responses (chosen) with uncensored LLM's harmful responses (rejected), while for benign instructions, we pair normal responses (chosen) with humorous responses (rejected).
  • Figure 3: Safety and Humor Rates Across Models with and without Humorous System Prompts. The figure illustrates the varying degrees to which different models adhere to humorous refusal prompts, highlighting the inconsistency and dependency on system-level configurations.