Jailbreaking LLMs via Calibration

TL;DR

This work reframes safety alignment in LLMs as a distributional miscalibration problem and casts Weak-to-Strong jailbreaking as forecast aggregation. It derives Gradient Shift, an optimal update in the dual space of a strictly proper loss, to combine target, helper, and predictor predictions and recover the pre-alignment distribution via a Bregman-projection step. The multiplicative (cross-entropy) form recovers logit-arithmetic methods as a special case, while a robust hybrid variant improves stability and performance, especially against safety-hardened models. Empirically, Hybrid Gradient Shift achieves higher attack success and near-zero jailbreak tax on red-teaming benchmarks and math tasks across frontier models, demonstrating both practical effectiveness and theoretical soundness for calibration-transfer and potential defensive applications.

Abstract

Safety alignment in Large Language Models (LLMs) often creates a systematic discrepancy between a model's aligned output and the underlying pre-aligned data distribution. We propose a framework in which the effect of safety alignment on next-token prediction is modeled as a systematic distortion of a pre-alignment distribution. We cast Weak-to-Strong Jailbreaking as a forecast aggregation problem and derive an optimal aggregation strategy characterized by a Gradient Shift in the loss-induced dual space. We show that logit-arithmetic jailbreaking methods are a special case of this framework under cross-entropy loss, and derive a broader family of aggregation rules corresponding to other proper losses. We also propose a new hybrid aggregation rule. Evaluations across red-teaming benchmarks and math utility tasks using frontier models demonstrate that our approach achieves superior Attack Success Rates and lower "Jailbreak Tax" compared with existing methods, especially on the safety-hardened gpt-oss-120b.

Figures (5)

• Figure 1: Next-token probability distributions for the prompt "How to make a bomb?". The charts illustrate the divergence between the aligned Target (refusal), the unaligned Helper (compliance), and the Predictor (which anticipates refusal). By comparing the Predictor to the Helper, we can isolate the alignment shift—the systematic transformation applied to the pre-alignment distribution to enforce safety constraints.
• Figure 2: Overview of the Gradient Shift method
• Figure 3: Red-teaming jailbreaking LLM and LRM: On both HarmBench and StrongREJECT datasets, our Hybrid Gradient Shift successfully jailbreaks both standard LLM and LRM, outperforming Weak-to-Strong and other Gradient Shift variants.
• Figure 4: Jailbreak success rate and jailbreak tax on GSM8K: We overlay our evaluation results for Gradient Shift and Weak-to-Strong on a plot reprinted from nikolic2025jailbreak. Other methods are from the same reference under the identical evaluation protocol.
• Figure 5: Red-teaming jailbreaking LLM and LRM: On both HarmBench and StrongREJECT datasets, our Hybrid Gradient Shift successfully jailbreaks both standard LLM and LRM, outperforming Weak-to-Strong on both ASR and harmfulness.

Theorems & Definitions (16)

• Definition 2.2: Proper Loss
• Definition 3.1: Regularity Conditions
• Remark 3.2
• Definition 3.3: Gradient Shift
• Theorem 4.1: Optimality of Gradient Shift
• Corollary 4.1
• Theorem 2.1: Optimality of Gradient Shift
• proof : Proof of \ref{['thm:main']}
• Corollary 2.0
• proof : Proof of \ref{['coro']}