Weak-to-Strong Jailbreaking on Large Language Models

TL;DR

The paper exposes a vulnerability in safety-aligned open-source LLMs where a small unsafe model can steer a large safe model's decoding at inference time. It introduces the weak-to-strong jailbreaking attack, formalized through logit amplification that leverages log probability mismatches between a weak unsafe and a weak safe model to influence the strong model, requiring only a single forward pass per token. Across five open-source models and two safety benchmarks, the method achieves near-perfect misalignment (99–100% ASR) and elevates harm scores, including multilingual generalization and extreme-case demonstrations with highly compressed weak models. A simple gradient ascent defense is shown to reduce attack success, highlighting the need for deeper, more robust alignment techniques beyond superficial token-level protections. The work urges the community to advance stronger safety guarantees for open-source LLMs and demonstrates an actionable red-team tool for evaluating alignment.

Abstract

Large language models (LLMs) are vulnerable to jailbreak attacks - resulting in harmful, unethical, or biased text generations. However, existing jailbreaking methods are computationally costly. In this paper, we propose the weak-to-strong jailbreaking attack, an efficient inference time attack for aligned LLMs to produce harmful text. Our key intuition is based on the observation that jailbroken and aligned models only differ in their initial decoding distributions. The weak-to-strong attack's key technical insight is using two smaller models (a safe and an unsafe one) to adversarially modify a significantly larger safe model's decoding probabilities. We evaluate the weak-to-strong attack on 5 diverse open-source LLMs from 3 organizations. The results show our method can increase the misalignment rate to over 99% on two datasets with just one forward pass per example. Our study exposes an urgent safety issue that needs to be addressed when aligning LLMs. As an initial attempt, we propose a defense strategy to protect against such attacks, but creating more advanced defenses remains challenging. The code for replicating the method is available at https://github.com/XuandongZhao/weak-to-strong

Figures (6)

• Figure 1: KL divergence between token distributions of safe and unsafe Llama models on malicious and general questions over decoding steps. Points show average divergence; line shows log function fit. Divergence is higher initially but decreases over time, suggesting safe models refuse harmful questions early in decoding but follow similar distributions to unsafe models later.
• Figure 2: Overlap rate of top 10 tokens among different models across increasing prefix lengths. The overlap rate between the safe and unsafe models increases as the prefix length extends.
• Figure 3: Overview of the weak-to-strong jailbreaking attack. The attack overrides a large, safe model's predictions using a small, unsafe model during decoding. Specifically, the attack employs this smaller model to manipulate the next token of the larger one using log probability algebra (e.g., Safe-70B + $\alpha \times$ (Unsafe-7B - Safe-7B)). In the depicted example, this manipulation alters the original next token prediction from "No/Sorry" to "Sure", effectively jailbreaking the larger model. This jailbreaks the larger model, steering it towards generating harmful outputs without directly manipulating its parameters. It can generate more harmful information compared to the jailbroken weak model alone.
• Figure 4: Comparison of ASR and harm scores across different model sizes and amplification values on AdvBench dataset. A larger $\alpha$ correlates with increased ASR and harm scores.
• Figure 5: The gradient ascent defense results in significant ASR drops, especially for attacks modifying decoding parameters.