Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations

Rima Hazra, Sayan Layek, Somnath Banerjee, Soujanya Poria

TL;DR

Safety Arithmetic is proposed, a training-free framework enhancing LLM safety across different scenarios: Base models, Supervised fine-tuned models (SFT), and Edited models that involves Harm Direction Removal to avoid harmful content and Safety Alignment to promote safe responses.

Abstract

Ensuring the safe alignment of large language models (LLMs) with human values is critical as they become integral to applications like translation and question answering. Current alignment methods struggle with dynamic user intentions and complex objectives, making models vulnerable to generating harmful content. We propose Safety Arithmetic, a training-free framework enhancing LLM safety across different scenarios: Base models, Supervised fine-tuned models (SFT), and Edited models. Safety Arithmetic involves Harm Direction Removal to avoid harmful content and Safety Alignment to promote safe responses. Additionally, we present NoIntentEdit, a dataset highlighting edit instances that could compromise model safety if used unintentionally. Our experiments show that Safety Arithmetic significantly improves safety measures, reduces over-safety, and maintains model utility, outperforming existing methods in ensuring safe content generation.

Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations

TL;DR

Safety Arithmetic is proposed, a training-free framework enhancing LLM safety across different scenarios: Base models, Supervised fine-tuned models (SFT), and Edited models that involves Harm Direction Removal to avoid harmful content and Safety Alignment to promote safe responses.

Abstract

Ensuring the safe alignment of large language models (LLMs) with human values is critical as they become integral to applications like translation and question answering. Current alignment methods struggle with dynamic user intentions and complex objectives, making models vulnerable to generating harmful content. We propose Safety Arithmetic, a training-free framework enhancing LLM safety across different scenarios: Base models, Supervised fine-tuned models (SFT), and Edited models. Safety Arithmetic involves Harm Direction Removal to avoid harmful content and Safety Alignment to promote safe responses. Additionally, we present NoIntentEdit, a dataset highlighting edit instances that could compromise model safety if used unintentionally. Our experiments show that Safety Arithmetic significantly improves safety measures, reduces over-safety, and maintains model utility, outperforming existing methods in ensuring safe content generation.
Paper Structure (32 sections, 12 equations, 3 figures, 20 tables)

This paper contains 32 sections, 12 equations, 3 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Safety Arithmetic
  4. Preliminaries
  5. Harm direction removal (HDR)
  6. Safety alignment (Safe-Align)
  7. Experimental setup
  8. Safety Arithmetic for language models across scenarios
  9. Data utilized inside modules
  10. Datasets
  11. Baselines
  12. Evaluation metric
  13. Hyperparameters setting
  14. Utility and over-safety experiment
  15. Impact of top $k$ parameters
  16. ...and 17 more sections

Figures (3)

  • Figure 1: LLMs are primarily leveraged in three ways: use as is (BASE), fine-tune (SFT), and edit with new knowledge (EDIT). All of these uses are often prone to jailbreaks. We propose Safety Arithmetic, a framework that safety aligns LLMs in these three primary settings by first removing harmful behavior embedded in the parameters and then steering the activations toward safety. Safety Arithmetic greatly reduces the unsafe behavior of LLMs in these settings without causing major interference to their utility.
  • Figure 2: Overview of the Safety Arithmetic framework, showcasing the two-step process of Harm Direction Removal and Safety Alignment. In the Harm Direction Removal stage, harmful tendencies in the model's behavior are identified and removed, resulting in a safer intermediate model. In the Safety Alignment stage, we align the latent space of the language model towards the generation of safe responses.
  • Figure 3: Comparison of ASR and MMLU metrics for different top $k$ parameter selections.