Reasoning over Precedents Alongside Statutes: Case-Augmented Deliberative Alignment for LLM Safety

TL;DR

The paper interrogates whether case-based precedents can outperform rule-based safety codes for aligning open-source LLMs. It finds that inference-time safety codes can harm helpfulness, while case-augmented reasoning improves harmlessness and robustness; training-time case-augmented reasoning via reinforcement learning (CADA) further enhances safety without sacrificing utility, even with limited data. By combining minimal safety codes with illustrative cases and a KL-penalized RL objective, CADA learns flexible, context-aware safety judgments from self-generated reasoning chains. The approach demonstrates superior Harmlessness and Robustness across multiple benchmarks and reduces over-refusal on benign queries, offering a practical alternative to code-heavy deliberative alignment in safety-critical deployments of open-source LLMs.

Abstract

Ensuring that Large Language Models (LLMs) adhere to safety principles without refusing benign requests remains a significant challenge. While OpenAI introduces deliberative alignment (DA) to enhance the safety of its o-series models through reasoning over detailed ``code-like'' safety rules, the effectiveness of this approach in open-source LLMs, which typically lack advanced reasoning capabilities, is understudied. In this work, we systematically evaluate the impact of explicitly specifying extensive safety codes versus demonstrating them through illustrative cases. We find that referencing explicit codes inconsistently improves harmlessness and systematically degrades helpfulness, whereas training on case-augmented simple codes yields more robust and generalized safety behaviors. By guiding LLMs with case-augmented reasoning instead of extensive code-like safety rules, we avoid rigid adherence to narrowly enumerated rules and enable broader adaptability. Building on these insights, we propose CADA, a case-augmented deliberative alignment method for LLMs utilizing reinforcement learning on self-generated safety reasoning chains. CADA effectively enhances harmlessness, improves robustness against attacks, and reduces over-refusal while preserving utility across diverse benchmarks, offering a practical alternative to rule-only DA for improving safety while maintaining helpfulness.

Figures (15)

• Figure 1: (a) Case study comparing model behaviors under SFT, DPO, and CADA on a context-dependent harmful request. (b) Safety and utility benchmark performance for LLaMA-3-8B and Deepthought-8B.
• Figure 2: Impact of inference-time safety codes on helpfulness and over-refusal. The Helpful Score and Non-Refusal Rate of LLaMA-3-8B/70B-Instruct and Deepthought-8B on the XSTest benchmark.
• Figure 3: Impact of training-time reasoning on helpfulness. The Helpful Score and Non-Refusal Rate of LLMs on the XSTest benchmark.
• Figure 4: Overview of CADA . In the Generation Phase, the model produces safety reasoning chains guided by minimal safety codes and illustrative cases. In the Optimization Phase, the policy is updated via RL, utilizing an CADA reward derived from response accuracy and reasoning quality to internalize adaptive safety judgments.
• Figure 5: Excerpt of the overall safety specification for all hazard categories.