Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements

TL;DR

This work rethinks safety alignment for large language models by introducing Controllable Safety Alignment (CoSA), which enables inference-time adaptation to diverse safety requirements without retraining. It achieves this through CoSAlign, a data-centric method that generates a rich set of safety configs from an explicit risk taxonomy, builds a labeled preference dataset via synthetic prompts and evaluators, and optimizes with supervised fine-tuning and Direct Preference Optimization to yield a controllable model $\mathcal{M}_{\text{ctrl}}$. The authors also present CoSApien, a real-world benchmark with five safety configurations and 200 prompts, and propose CoSA-Score as a unified metric for measuring controllability of safety and helpfulness. Experimental results show that CoSAlign substantially improves controllability over strong baselines (ICA and cascade-based methods) and generalizes to unseen safety configs, while maintaining general capability and safety. The framework emphasizes allow-listing access to authorized users and a config-review process to ensure safe deployment, highlighting practical considerations for pluralistic safety in real-world AI systems.

Abstract

The current paradigm for safety alignment of large language models (LLMs) follows a one-size-fits-all approach: the model refuses to interact with any content deemed unsafe by the model provider. This approach lacks flexibility in the face of varying social norms across cultures and regions. In addition, users may have diverse safety needs, making a model with static safety standards too restrictive to be useful, as well as too costly to be re-aligned. We propose Controllable Safety Alignment (CoSA), a framework designed to adapt models to diverse safety requirements without re-training. Instead of aligning a fixed model, we align models to follow safety configs -- free-form natural language descriptions of the desired safety behaviors -- that are provided as part of the system prompt. To adjust model safety behavior, authorized users only need to modify such safety configs at inference time. To enable that, we propose CoSAlign, a data-centric method for aligning LLMs to easily adapt to diverse safety configs. Furthermore, we devise a novel controllability evaluation protocol that considers both helpfulness and configured safety, summarizing them into CoSA-Score, and construct CoSApien, a human-authored benchmark that consists of real-world LLM use cases with diverse safety requirements and corresponding evaluation prompts. We show that CoSAlign leads to substantial gains of controllability over strong baselines including in-context alignment. Our framework encourages better representation and adaptation to pluralistic human values in LLMs, and thereby increasing their practicality.

Figures (7)

• Figure 1: Controllable safety alignment (1) produces a single LLM with controllable safety using our proposed CoSAlign method; (2) incorporates safety configs from authorized users into the LLM and returns a custom interface for each user, allowing users with different safety needs to be served without re-training.
• Figure 2: Left: Illustration of the config review process. Given a config $s_i$ from the user, the model provider first reviews the config and makes necessary change, producing $s_i'$. Finally, the custom interface $\mathcal{M}_\text{ctrl}(\cdot | s_i')$ is returned to the user. Right: Illustration of the evaluation protocol.
• Figure 3: ICA v.s. CoSAlign on CoSAlign-Test. ICA is ineffective under complex safety configs.
• Figure 4: ICA of GPT-4o-mini deteriorates CoSA-Score on CoSAlign-Test.
• Figure 5: CoSAlign pipeline for synthesizing categorical safety configs.