Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness

TL;DR

This work investigates how to align instruction-tuned language models toward safety and usefulness. By combining safety-focused instruction data (SIT), reward-based fine-tuning (RAFT), and direct preference optimization (DPO) on safe RLHF data, the authors demonstrate substantial reductions in unsafe outputs while preserving or enhancing helpfulness, with DPO delivering the strongest performance. The study provides a rigorous evaluation framework spanning helpfulness and harmfulness metrics, plus analyses of modelError modes to guide future alignment strategies. The findings have practical implications for deploying safer LLMs in real-world settings, highlighting the importance of safety signals during instruction tuning and the value of learning from both selections and rejections of model outputs.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning and text generation. However, these models can inadvertently generate unsafe or biased responses when prompted with problematic inputs, raising significant ethical and practical concerns for real-world deployment. This research addresses the critical challenge of developing language models that generate both helpful and harmless content, navigating the delicate balance between model performance and safety. We demonstrate that incorporating safety-related instructions during the instruction-tuning of pre-trained models significantly reduces toxic responses to unsafe prompts without compromising performance on helpfulness datasets. We found Direct Preference Optimization (DPO) to be particularly effective, outperforming both SIT and RAFT by leveraging both chosen and rejected responses for learning. Our approach increased safe responses from 40$\%$ to over 90$\%$ across various harmfulness benchmarks. In addition, we discuss a rigorous evaluation framework encompassing specialized metrics and diverse datasets for safety and helpfulness tasks ensuring a comprehensive assessment of the model's capabilities.

Figures (12)

• Figure 1: Instruction-Template
• Figure 2: Instruction-Template with no Input
• Figure 3: Basemodel training (Left). Safety Instruction tuning training (Right)
• Figure 4: SIT Evaluation Results - Llama guard safe % (Left), Helpfulness QA Tasks + Single-Word Answers (Right)
• Figure 5: RAFT - Llama Guard Safe % - Different RM (left) and Different Iterations (right)