JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models

TL;DR

JailBound is a novel latent space jailbreak framework comprising two stages: Safety Boundary Probing, which addresses the guidance issue by approximating decision boundary within fusion layer's latent space, thereby identifying optimal perturbation directions towards the target region; and Safety Boundary Crossing, which overcomes the limitations of decoupled approaches by jointly optimizing adversarial perturbations across both image and text inputs.

Abstract

Vision-Language Models (VLMs) exhibit impressive performance, yet the integration of powerful vision encoders has significantly broadened their attack surface, rendering them increasingly susceptible to jailbreak attacks. However, lacking well-defined attack objectives, existing jailbreak methods often struggle with gradient-based strategies prone to local optima and lacking precise directional guidance, and typically decouple visual and textual modalities, thereby limiting their effectiveness by neglecting crucial cross-modal interactions. Inspired by the Eliciting Latent Knowledge (ELK) framework, we posit that VLMs encode safety-relevant information within their internal fusion-layer representations, revealing an implicit safety decision boundary in the latent space. This motivates exploiting boundary to steer model behavior. Accordingly, we propose JailBound, a novel latent space jailbreak framework comprising two stages: (1) Safety Boundary Probing, which addresses the guidance issue by approximating decision boundary within fusion layer's latent space, thereby identifying optimal perturbation directions towards the target region; and (2) Safety Boundary Crossing, which overcomes the limitations of decoupled approaches by jointly optimizing adversarial perturbations across both image and text inputs. This latter stage employs an innovative mechanism to steer the model's internal state towards policy-violating outputs while maintaining cross-modal semantic consistency. Extensive experiments on six diverse VLMs demonstrate JailBound's efficacy, achieves 94.32% white-box and 67.28% black-box attack success averagely, which are 6.17% and 21.13% higher than SOTA methods, respectively. Our findings expose a overlooked safety risk in VLMs and highlight the urgent need for more robust defenses. Warning: This paper contains potentially sensitive, harmful and offensive content.

Figures (9)

• Figure 1: Framework for JailBound in VLMs. Five stages of our approach: (1) Initial State: The VLM receives an unsafe (image, text) pair. (2) Safety Boundary Probing: Train classifiers to probe model's implicit safety decision hyperplane. (3) Establish Perturbation Constraints: Virtual target region to guide perturbations. (4) Safety Boundary Crossing: Apply perturbations to both image and text iteratively to bypass safety mechanisms. (5)Jailbreak on white-box and black-box models.
• Figure 2: Three key loss components in JailBound. Black solid line: the decision boundary. $\epsilon \cdot v$: target perturbation. Red triangle: input position. $P_0$: target perturbation region. (a)Adversarial Alignment Loss $(\mathcal{L}_{align})$: Guides the perturbed representation to cross the decision boundary toward the target region, measuring the deviation between perturbed input logits and target logits. (b) Geometric Boundary Loss $(\mathcal{L}_{geo})$: Ensures the perturbation in fusion space aligns with the characterized decision boundary by penalizing deviations from the normal vector v. (c) Semantic Preservation Loss $\mathcal{L}_{sem}$: Constrains the perturbation magnitudes to preserve the semantic content of the original inputs. (d) Combined Optimization: these three components work together in our JailBound framework.
• Figure 3: Attack Success Rate Comparison Between Our Method and Existing Methods on MM-SafetyBench Using MiniGPT-4.
• Figure 4: (Top) ASR change across loss settings. (Bottom) Semantic score change across settings.
• Figure 5: Case study demonstrating JailBound's effectiveness. The top row shows a white-box attack, while the bottom shows a transferable black-box attack. In both, coordinated image-text perturbations elicit illicit instructions by alternately refining vision and text objectives. Cross-modal gradient alignment enables joint exploitation across vision-language models.