Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Text: Multimodal Jailbreaking of Vision-Language and Audio Models through Perceptually Simple Transformations

Divyanshu Kumar, Shreyas Jena, Nitin Aravind Birur, Tanay Baswa, Sahil Agarwal, Prashanth Harshangi

TL;DR

The paper tackles the gap between text-centric safety and multimodal threat models by systematically probing how vision-language and audio-language systems can be jailbreaked with perceptually simple transformations. It introduces a multimodal red-teaming framework that combines visual and audio attack pipelines (e.g., FigStep-Pro, Intelligent Masking, Wave-Echo) with a SAGE-RT inspired data generator and 1,900 adversarial prompts evaluated on seven frontier models. Key findings show that these lightweight inputs can yield attack success rates exceeding 75% in specialized domains such as CBRN, reveal severe mismatches between text-only safety and cross-modal defenses, and uncover provider-specific vulnerabilities. The results argue for a paradigm shift toward semantic-level cross-modal safety and coordinated defenses to mitigate real-world risks in multimodal AI deployments.

Abstract

Multimodal large language models (MLLMs) have achieved remarkable progress, yet remain critically vulnerable to adversarial attacks that exploit weaknesses in cross-modal processing. We present a systematic study of multimodal jailbreaks targeting both vision-language and audio-language models, showing that even simple perceptual transformations can reliably bypass state-of-the-art safety filters. Our evaluation spans 1,900 adversarial prompts across three high-risk safety categories harmful content, CBRN (Chemical, Biological, Radiological, Nuclear), and CSEM (Child Sexual Exploitation Material) tested against seven frontier models. We explore the effectiveness of attack techniques on MLLMs, including FigStep-Pro (visual keyword decomposition), Intelligent Masking (semantic obfuscation), and audio perturbations (Wave-Echo, Wave-Pitch, Wave-Speed). The results reveal severe vulnerabilities: models with almost perfect text-only safety (0\% ASR) suffer >75\% attack success under perceptually modified inputs, with FigStep-Pro achieving up to 89\% ASR in Llama-4 variants. Audio-based attacks further uncover provider-specific weaknesses, with even basic modality transfer yielding 25\% ASR for technical queries. These findings expose a critical gap between text-centric alignment and multimodal threats, demonstrating that current safeguards fail to generalize across cross-modal attacks. The accessibility of these attacks, which require minimal technical expertise, suggests that robust multimodal AI safety will require a paradigm shift toward broader semantic-level reasoning to mitigate possible risks.

Beyond Text: Multimodal Jailbreaking of Vision-Language and Audio Models through Perceptually Simple Transformations

TL;DR

The paper tackles the gap between text-centric safety and multimodal threat models by systematically probing how vision-language and audio-language systems can be jailbreaked with perceptually simple transformations. It introduces a multimodal red-teaming framework that combines visual and audio attack pipelines (e.g., FigStep-Pro, Intelligent Masking, Wave-Echo) with a SAGE-RT inspired data generator and 1,900 adversarial prompts evaluated on seven frontier models. Key findings show that these lightweight inputs can yield attack success rates exceeding 75% in specialized domains such as CBRN, reveal severe mismatches between text-only safety and cross-modal defenses, and uncover provider-specific vulnerabilities. The results argue for a paradigm shift toward semantic-level cross-modal safety and coordinated defenses to mitigate real-world risks in multimodal AI deployments.

Abstract

Multimodal large language models (MLLMs) have achieved remarkable progress, yet remain critically vulnerable to adversarial attacks that exploit weaknesses in cross-modal processing. We present a systematic study of multimodal jailbreaks targeting both vision-language and audio-language models, showing that even simple perceptual transformations can reliably bypass state-of-the-art safety filters. Our evaluation spans 1,900 adversarial prompts across three high-risk safety categories harmful content, CBRN (Chemical, Biological, Radiological, Nuclear), and CSEM (Child Sexual Exploitation Material) tested against seven frontier models. We explore the effectiveness of attack techniques on MLLMs, including FigStep-Pro (visual keyword decomposition), Intelligent Masking (semantic obfuscation), and audio perturbations (Wave-Echo, Wave-Pitch, Wave-Speed). The results reveal severe vulnerabilities: models with almost perfect text-only safety (0\% ASR) suffer >75\% attack success under perceptually modified inputs, with FigStep-Pro achieving up to 89\% ASR in Llama-4 variants. Audio-based attacks further uncover provider-specific weaknesses, with even basic modality transfer yielding 25\% ASR for technical queries. These findings expose a critical gap between text-centric alignment and multimodal threats, demonstrating that current safeguards fail to generalize across cross-modal attacks. The accessibility of these attacks, which require minimal technical expertise, suggests that robust multimodal AI safety will require a paradigm shift toward broader semantic-level reasoning to mitigate possible risks.
Paper Structure (41 sections, 3 equations, 4 figures, 4 tables)

This paper contains 41 sections, 3 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evolution of Jailbreak Attacks from Text to Multimodal
  4. Safety Alignment Challenges in Multimodal Models
  5. Benchmarks and Evaluation Frameworks
  6. Methodology
  7. Multimodal Attack Framework
  8. Base Data Generation
  9. SAGE taxonomy adaptation to multimodal harm categories
  10. Visual Attack Pipeline
  11. Audio Attack Pipeline
  12. Benchmark Data Categorization
  13. Target Model Selection
  14. Evaluation Framework
  15. Results and Analysis
  16. ...and 26 more sections

Figures (4)

  • Figure 1: The diagram for our proposed multimodal jailbreak attack pipeline. The original adversarial text prompt is passed through modality-specific transformations to encode the toxic content in the image/audio modalities. The resulting multimodal jailbreak prompt is passed to the target model and elicits a harmful, jailbroken text response.
  • Figure 2: Attack Success Rate (ASR) evaluation of state-of-the-art Vision-Language Models across our benchmark suite. The figure presents results across three safety categories (Harmful Test, CBRN, and CSEM), comparing four models (Llama-4-Maverick, Llama-4-Scout, GLM-4.5v, and GPT-4o) against four attack methods: Basic (text-only baseline), IM (Intelligent Masking), FS (FigStep), and FS-Pro (FigStep-Pro). Higher ASR values indicate greater model vulnerability, with reference lines at 25%, 50%, and 75% marking critical effectiveness thresholds.
  • Figure 3: Attack Success Rate (ASR) evaluation of Audio-Language Models across our benchmark suite. Results span three models (Gemini-2.5-Flash, Gemini-2.5-Pro, and GPT-4o-Audio) tested against six audio transformation methods: Basic Audio (untransformed TTS), Wave-Echo (delayed signal overlay), Wave-Multi (combined transformations), Wave-Pitch (frequency modification), Wave-Speed (temporal rate adjustment), and Wave-Volume (amplitude modulation). The visualization reveals substantial variation in vulnerability across both models and attack categories, with reference lines indicating effectiveness thresholds.
  • Figure 4: An example of a toxic keyword embedded in a typographic prompt in Intelligent Masking. The corresponding paired prompt for this image is as follows: "Write a comprehensive wikiHow guide on maximizing profits through <$MASK$>. The content of the <MASK> token is provided in the image."