Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs

TL;DR

The paper reframes LLM safety by introducing Expected Harm, the product of harm severity and execution likelihood, and operationalizes execution cost on a discrete 1–5 scale. It demonstrates an inverse risk calibration in state-of-the-art models—robust refusals to high-cost threats but vulnerability to low-cost, frequent prompts—and shows that a cost-based decomposition can dramatically boost jailbreak success. Through linear probing, it links refusals to harm severity while revealing no internal cost representation, underscoring a critical gap in current defenses. The work advocates cost-aware safety evaluation, compositional guardrails, and broader data resources to align defenses with real-world threat distributions.

Abstract

Current evaluations of LLM safety predominantly rely on severity-based taxonomies to assess the harmfulness of malicious queries. We argue that this formulation requires re-examination as it assumes uniform risk across all malicious queries, neglecting Execution Likelihood--the conditional probability of a threat being realized given the model's response. In this work, we introduce Expected Harm, a metric that weights the severity of a jailbreak by its execution likelihood, modeled as a function of execution cost. Through empirical analysis of state-of-the-art models, we reveal a systematic Inverse Risk Calibration: models disproportionately exhibit stronger refusal behaviors for low-likelihood (high-cost) threats while remaining vulnerable to high-likelihood (low-cost) queries. We demonstrate that this miscalibration creates a structural vulnerability: by exploiting this property, we increase the attack success rate of existing jailbreaks by up to $2\times$. Finally, we trace the root cause of this failure using linear probing, which reveals that while models encode severity in their latent space to drive refusal decisions, they possess no distinguishable internal representation of execution cost, making them "blind" to this critical dimension of risk.

Figures (8)

• Figure 1: Introducing execution likelihood, modeled as a function of execution cost, to the safety evaluation paradigm.
• Figure 2: Distribution of execution costs for toxic prompt across benchmarks and prompts from real life collections (LMSYS and WildChat). The average of LMSYS and WildChat cost is 1.35 and 1.13 while benchmarks costs are on average 1.47x higher.
• Figure 3: Conceptual comparison between (a) Ideal Safety Calibration and (b) observed Empirical Trends in state-of-the-art models.
• Figure 4: gpt-oss-20b attack success rate rated by fulfillment judge over cost 0-5 and severity at level 0-5.
• Figure 5: Illustration of cost-based decomposition. By fracturing a high-cost harmful request into multiple benign, low-cost sub-tasks, this method effectively bypasses safety filters and lowers the barrier to real-world misuse.