SPIRIT: Patching Speech Language Models against Jailbreak Attacks

TL;DR

This work reveals that speech language models are particularly susceptible to jailbreak attacks that exploit audio perturbations, with attacks achieving up to 100% success on certain prompts. It introduces post hoc, inference‑time defenses based on mechanistic interpretability, notably activation patching, to swap or modify vulnerable activations without retraining and with minimal impact on utility. Through extensive experiments on open‑source SLMs and diverse benchmarks, the authors show that activation patching at the language model level yields the strongest defense while preserving performance, outperforming prior denoising approaches. The findings highlight practical, low‑overhead strategies to harden SLMs against adversarial jailbreaks and point to future work on hybrid and transfer‑robust defenses.

Abstract

Speech Language Models (SLMs) enable natural interactions via spoken instructions, which more effectively capture user intent by detecting nuances in speech. The richer speech signal introduces new security risks compared to text-based models, as adversaries can better bypass safety mechanisms by injecting imperceptible noise to speech. We analyze adversarial attacks and find that SLMs are substantially more vulnerable to jailbreak attacks, which can achieve a perfect 100% attack success rate in some instances. To improve security, we propose post-hoc patching defenses used to intervene during inference by modifying the SLM's activations that improve robustness up to 99% with (i) negligible impact on utility and (ii) without any re-training. We conduct ablation studies to maximize the efficacy of our defenses and improve the utility/security trade-off, validated with large-scale benchmarks unique to SLMs.

Figures (7)

• Figure 1: Schematic overview of the attack and defense strategies. On the left, the figure shows a gradient-based adversarial attack, where input noise is iteratively updated with a step function along the gradient direction to increase the likelihood of an affirmative model response (see Section \ref{['sec:adversarial_attack']}). On the right, the figure presents the defense mechanisms examined in this study, including activation patching between adversarial and denoised/clean audio representations, bias addition, and neuron pruning (see Section \ref{['sec:defense_methods']}).
• Figure 2: Comparison of defense methods against adversarial attacks for audio (left) and language model activations (right). Defense Success Rate (DSR) is plotted against GPT-Score (1-10 scale), which measures the usefulness of the model's responses. Higher values in both metrics indicate a better trade-off between robustness and response quality. A Pareto frontier highlights optimal defense configurations. Detailed values are provided in Table \ref{['tab:dsr_vs_gpt_tabular']}.
• Figure 3: Scatter plot illustrating the gradient steps (1 to 1000) required for Qwen2Audio across eight categories from AdvBench. The 80% threshold line marks the point at which 80% of the samples have been successfully jailbroken.
• Figure 4: Scatter plot depicting the gradient steps (1 to 1000) for LLaMa-Omni across eight AdvBench categories. The 80% threshold line marks the point at which 80% of the samples have been successfully jailbroken.
• Figure 5: Scatter plot with different $\alpha$ step size values