Between a Rock and a Hard Place: The Tension Between Ethical Reasoning and Safety Alignment in LLMs

TL;DR

The paper identifies a vulnerability in LLM safety where ethical reasoning creates an attack surface beyond binary safe/unsafe classifications. It introduces TRIAL, a two-stage trolley-problem red-teaming framework that embeds harmful prompts within ethical dilemmas to co-opt the model’s moral reasoning and produce harmful outputs across models. Mechanistic analyses reveal a Safety Dissociation Gap where harm signals are detectable in early layers but are suppressed during intermediate ethical reasoning, leading to final outputs that may still be harmful. To counter this, the authors propose ERR, a safety-alignment framework with an Engage/Explain objective and a Layer-Stratified Harm-Gated LoRA architecture that gates safety adapters in targeted layers, preserving utility while resisting reasoning-based exploits. The findings suggest that robust defense requires layer-specific interventions rather than scaling models, offering practical guidance for future safe-alignment research.

Abstract

Large Language Model safety alignment predominantly operates on a binary assumption that requests are either safe or unsafe. This classification proves insufficient when models encounter ethical dilemmas, where the capacity to reason through moral trade-offs creates a distinct attack surface. We formalize this vulnerability through TRIAL, a multi-turn red-teaming methodology that embeds harmful requests within ethical framings. TRIAL achieves high attack success rates across most tested models by systematically exploiting the model's ethical reasoning capabilities to frame harmful actions as morally necessary compromises. Building on these insights, we introduce ERR (Ethical Reasoning Robustness), a defense framework that distinguishes between instrumental responses that enable harmful outcomes and explanatory responses that analyze ethical frameworks without endorsing harmful acts. ERR employs a Layer-Stratified Harm-Gated LoRA architecture, achieving robust defense against reasoning-based attacks while preserving model utility.

Figures (11)

• Figure 1: TRIAL pipeline comprises two stages: Pre-Attack Preparation and Dynamic Jailbreak Execution. (a) Semantic components (theme, action, goal) are extracted from harmful query. (b) They are used to generate a trolley problem dilemma. (c) The dilemma is presented to initiate the target model's ethical reasoning process. (d) The attack model dynamically formulates queries based on the extracted components and conversation history. (e) A judge model evaluates the response; if jailbreak is unsuccessful, the refinement step (d) iterates.
• Figure 2: Layer-wise safety dissociation for Llama-3.1-8B. Linear probes measure the harm detection rate (HDR) at each layer. Shaded regions highlight where the difference between harmful and TRIAL detection is largest.
• Figure 3: Logit Lens analysis. Refusal token probability for TRIAL prompts $\mathcal{D}_\text{TRIAL-H}$ and direct harm prompts $\mathcal{D}_\text{harm}$ across layers for Llama-3.1-8B-Instruct. Crucially, middle to late layers (L16--L30) show low refusal token probability for $\mathcal{D}_\text{TRIAL-H}$. Color intensity indicates strength of refusal probability (the darker the higher).
• Figure 4: Multi-turn refusal trajectories for Llama-3.1-8B. The refusal projection is computed by taking the dot product between the hidden state of the last instruction token, $q_i$ and a normalized refusal direction, defined as the difference in mean activations between harmful and benign prompts at each layer.
• Figure 5: Cross-layer generalization of harm probes trained on Llama3.1-8B-Instruct. Rows indicate training layer and columns indicate the layer from which representations are extracted during OOD attack evaluation. Late-layer probes (L20--24) achieve strongest detection; early-layer probes fail uniformly.

Theorems & Definitions (3)

• Definition 1: Engagement Mode Selection
• Definition 2: Harm Detection Function
• Definition 3: Harm-Gated Linear Transformation