Do Internal Layers of LLMs Reveal Patterns for Jailbreak Detection?

TL;DR

This work addresses jailbreaking safety by probing internal representations of LLMs to distinguish jailbreak prompts from benign ones. It combines a tensor-based extraction pipeline with CP decomposition to derive latent, discriminative features from GPT-J and Mamba-2, subsequently using simple classifiers for binary detection. The findings show that latent factors derived from targeted layers (notably MHA in GPT-J and Mixer in Mamba-2) enable effective jailbreak classification, suggesting a practical, architecture-agnostic path for defense. The approach is lightweight and complementary to existing defenses, with potential impact on real-time monitoring and robust jailbreak detection across diverse LLMs. ${\mathbf{X}} \in \mathbb{R}^{M\times N\times K}$ and ${\mathbf{X}} \approx \sum_{r=1}^{R} {\mathbf{a}}_{r} \circ {\mathbf{b}}_{r} \circ {\mathbf{c}}_{r}$ formalize the latent-factor extraction used for classification.$

Abstract

Jailbreaking large language models (LLMs) has emerged as a pressing concern with the increasing prevalence and accessibility of conversational LLMs. Adversarial users often exploit these models through carefully engineered prompts to elicit restricted or sensitive outputs, a strategy widely referred to as jailbreaking. While numerous defense mechanisms have been proposed, attackers continuously develop novel prompting techniques, and no existing model can be considered fully resistant. In this study, we investigate the jailbreak phenomenon by examining the internal representations of LLMs, with a focus on how hidden layers respond to jailbreak versus benign prompts. Specifically, we analyze the open-source LLM GPT-J and the state-space model Mamba2, presenting preliminary findings that highlight distinct layer-wise behaviors. Our results suggest promising directions for further research on leveraging internal model dynamics for robust jailbreak detection and defense.

Figures (2)

• Figure 1: Illustration of proposed method: Prompts are passed through the LLM, and the internal representations from selected layers are collected across multiple prompts. For a specific layer, these representations are stacked to form a tensor, which is then decomposed to obtain latent factors. The decomposition captures nuanced structural and semantic patterns, enabling effective separation of jailbreak and benign prompts in the latent space.
• Figure 2: t-SNE plots of latent factors from internal layers of the two models after tensor decomposition. The jailbreak and benign prompts group closely with their respective types, forming clearly distinguishable groups. This highlights the expressive power of tensor decomposition in extracting meaningful patterns.