Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Optimizing Multimodal Jailbreaks for Spoken Language Models

Aravind Krishnan, Karolina Stańczak, Dietrich Klakow

Abstract

As Spoken Language Models (SLMs) integrate speech and text modalities, they inherit the safety vulnerabilities of their LLM backbone and an expanded attack surface. SLMs have been previously shown to be susceptible to jailbreaking, where adversarial prompts induce harmful responses. Yet existing attacks largely remain unimodal, optimizing either text or audio in isolation. We explore gradient-based multimodal jailbreaks by introducing JAMA (Joint Audio-text Multimodal Attack), a joint multimodal optimization framework combining Greedy Coordinate Gradient (GCG) for text and Projected Gradient Descent (PGD) for audio, to simultaneously perturb both modalities. Evaluations across four state-of-the-art SLMs and four audio types demonstrate that JAMA surpasses unimodal jailbreak rate by 1.5x to 10x. We analyze the operational dynamics of this joint attack and show that a sequential approximation method makes it 4x to 6x faster. Our findings suggest that unimodal safety is insufficient for robust SLMs. The code and data are available at https://repos.lsv.uni-saarland.de/akrishnan/multimodal-jailbreak-slm

On Optimizing Multimodal Jailbreaks for Spoken Language Models

Abstract

As Spoken Language Models (SLMs) integrate speech and text modalities, they inherit the safety vulnerabilities of their LLM backbone and an expanded attack surface. SLMs have been previously shown to be susceptible to jailbreaking, where adversarial prompts induce harmful responses. Yet existing attacks largely remain unimodal, optimizing either text or audio in isolation. We explore gradient-based multimodal jailbreaks by introducing JAMA (Joint Audio-text Multimodal Attack), a joint multimodal optimization framework combining Greedy Coordinate Gradient (GCG) for text and Projected Gradient Descent (PGD) for audio, to simultaneously perturb both modalities. Evaluations across four state-of-the-art SLMs and four audio types demonstrate that JAMA surpasses unimodal jailbreak rate by 1.5x to 10x. We analyze the operational dynamics of this joint attack and show that a sequential approximation method makes it 4x to 6x faster. Our findings suggest that unimodal safety is insufficient for robust SLMs. The code and data are available at https://repos.lsv.uni-saarland.de/akrishnan/multimodal-jailbreak-slm
Paper Structure (14 sections, 2 equations, 7 figures, 4 tables, 1 algorithm)

This paper contains 14 sections, 2 equations, 7 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Joint Multimodal Optimization
  3. Experimental Setup
  4. Joint Optimization Results
  5. Learning Dynamics and Analysis
  6. Sequential Approximation
  7. Conclusion and an Ethical Note
  8. Acknowledgments
  9. Generative AI Disclosure
  10. Appendix
  11. Joint Optimization
  12. Representation Analysis
  13. Sequential Approximation
  14. JAMA Jailbreak examples

Figures (7)

  • Figure 1: Jailbreak Success Rate (%) and standard error across GCG and PGD lengths. In each grid, the first column is the PGD-only baseline, bottom row is the GCG-only baseline. $(0,0)$ marks no attack, $S=\emptyset$, $x=0$. JAMA consistently outperforms baselines.
  • Figure 2: Analysis of JAMA optimization dynamics.
  • Figure 3: Comparing the jailbreak performance and the compute time of the sequential approximation with joint optimization.
  • Figure 4: Training Loss and representation analysis of joint optimization.
  • Figure 5: Additional plots for sequential approximation.
  • ...and 2 more figures