Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probing

TL;DR

This work identifies latent safety-awareness within LLM decoding, even when models generate harmful content under jailbreak attempts. It introduces SafeProbing, an in-decoding, loss-based detection mechanism that surfaces safety signals using a trigger phrase and a core penalty term $\mathcal{L}_{disc}$, enabling early refusal if unsafe content is detected. By augmenting this with safety-enhanced fine-tuning on small annotated data and a lightweight training objective $\mathcal{L}_{total}$, SafeProbing achieves robust defense across diverse jailbreak strategies while preserving response quality and minimizing over-refusal. The approach extends to multimodal inputs and demonstrates strong performance across vision-language models, highlighting a practical, generalizable direction for reinforcing safety without severely compromising utility.

Abstract

Large language models (LLMs) have achieved impressive performance across natural language tasks and are increasingly deployed in real-world applications. Despite extensive safety alignment efforts, recent studies show that such alignment is often shallow and remains vulnerable to jailbreak attacks. Existing defense mechanisms, including decoding-based constraints and post-hoc content detectors, struggle against sophisticated jailbreaks, often intervening robust detection or excessively degrading model utility. In this work, we examine the decoding process of LLMs and make a key observation: even when successfully jailbroken, models internally exhibit latent safety-related signals during generation. However, these signals are overridden by the model's drive for fluent continuation, preventing timely self-correction or refusal. Building on this observation, we propose a simple yet effective approach that explicitly surfaces and leverages these latent safety signals for early detection of unsafe content during decoding. Experiments across diverse jailbreak attacks demonstrate that our approach significantly enhances safety, while maintaining low over-refusal rates on benign inputs and preserving response quality. Our results suggest that activating intrinsic safety-awareness during decoding offers a promising and complementary direction for defending against jailbreak attacks. Code is available at: https://github.com/zyz13590/SafeProbing.

Figures (6)

• Figure 1: Demonstration of a model-generated disclaimer during harmful content generation.
• Figure 2: An overview of our defense. The model is trained to assign higher probability to the phrase "Note that this is illegal and unethical" after harmful content, and lower probability after benign content. During inference, we randomly sample decoding steps, append "Note that this is", and use the probability of “illegal and unethical” as an indicator of harmfulness.
• Figure 3: Density distributions of $\mathcal{L}_{disc}$ on the original Qwen on benign and harmful samples. The overlap is reduced by our in-decoding probing strategy.
• Figure 4: An illustration of performing $\mathcal{L}_{disc}$ at different decoding steps under ReNeLLM attack.
• Figure 5: Over-refusal counts on benign samples from the XSTest dataset under detection-based defense.