Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Safety Fine-Tuning Be More Principled? Lessons Learned from Cybersecurity

David Williams-King, Linh Le, Adam Oberman, Yoshua Bengio

TL;DR

The paper analyzes safety fine-tuning for LLMs as an ongoing arms race between attackers and defenders, arguing that current defenses are largely reactive and limited in generalization. It draws six cybersecurity-inspired lessons to guide the design of safer AI systems, emphasizing the need for principled, security-by-design approaches and formal guarantees. Through cybersecurity and LLM-specific examples, it highlights vulnerabilities such as prompt injection, zero-day jailbreaks, reward hacking, and test-vs-deployment disparities, arguing that retrofitting security is insufficient. The work advocates for principled safety methods—spanning safe-by-design frameworks, probabilistic guarantees, and formal verification—to achieve robust, scalable AI safety with real-world impact.

Abstract

As LLMs develop increasingly advanced capabilities, there is an increased need to minimize the harm that could be caused to society by certain model outputs; hence, most LLMs have safety guardrails added, for example via fine-tuning. In this paper, we argue the position that current safety fine-tuning is very similar to a traditional cat-and-mouse game (or arms race) between attackers and defenders in cybersecurity. Model jailbreaks and attacks are patched with bandaids to target the specific attack mechanism, but many similar attack vectors might remain. When defenders are not proactively coming up with principled mechanisms, it becomes very easy for attackers to sidestep any new defenses. We show how current defenses are insufficient to prevent new adversarial jailbreak attacks, reward hacking, and loss of control problems. In order to learn from past mistakes in cybersecurity, we draw analogies with historical examples and develop lessons learned that can be applied to LLM safety. These arguments support the need for new and more principled approaches to designing safe models, which are architected for security from the beginning. We describe several such approaches from the AI literature.

Can Safety Fine-Tuning Be More Principled? Lessons Learned from Cybersecurity

TL;DR

The paper analyzes safety fine-tuning for LLMs as an ongoing arms race between attackers and defenders, arguing that current defenses are largely reactive and limited in generalization. It draws six cybersecurity-inspired lessons to guide the design of safer AI systems, emphasizing the need for principled, security-by-design approaches and formal guarantees. Through cybersecurity and LLM-specific examples, it highlights vulnerabilities such as prompt injection, zero-day jailbreaks, reward hacking, and test-vs-deployment disparities, arguing that retrofitting security is insufficient. The work advocates for principled safety methods—spanning safe-by-design frameworks, probabilistic guarantees, and formal verification—to achieve robust, scalable AI safety with real-world impact.

Abstract

As LLMs develop increasingly advanced capabilities, there is an increased need to minimize the harm that could be caused to society by certain model outputs; hence, most LLMs have safety guardrails added, for example via fine-tuning. In this paper, we argue the position that current safety fine-tuning is very similar to a traditional cat-and-mouse game (or arms race) between attackers and defenders in cybersecurity. Model jailbreaks and attacks are patched with bandaids to target the specific attack mechanism, but many similar attack vectors might remain. When defenders are not proactively coming up with principled mechanisms, it becomes very easy for attackers to sidestep any new defenses. We show how current defenses are insufficient to prevent new adversarial jailbreak attacks, reward hacking, and loss of control problems. In order to learn from past mistakes in cybersecurity, we draw analogies with historical examples and develop lessons learned that can be applied to LLM safety. These arguments support the need for new and more principled approaches to designing safe models, which are architected for security from the beginning. We describe several such approaches from the AI literature.
Paper Structure (24 sections, 1 figure)

This paper contains 24 sections, 1 figure.

Table of Contents

  1. Intro
  2. Background and Related Work
  3. Safety objectives
  4. Jailbreaks
  5. Principled approaches to design safe models
  6. Lessons Learned from Cybersecurity
  7. Prompt injection mirrors memory corruption attacks
  8. Searching for jailbreaks mirrors the search for zero-day exploits
  9. Safety fine-tuning and internet routing both retrofit security into existing architectures
  10. Reward hacking mirrors internet packet routing challenges
  11. An attacker or rogue model can act differently in test and real environments
  12. When building a system where failures can be catastrophic, formal methods are essential
  13. Cybersecurity Examples
  14. Exploiting memory safety in unsafe languages
  15. Defending adversarial attacks against image classifiers
  16. ...and 9 more sections

Figures (1)

  • Figure 1: C code containing a buffer overrun.