Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MUSE: A Run-Centric Platform for Multimodal Unified Safety Evaluation of Large Language Models

Zhongxi Wang, Yueqian Lin, Jingyang Zhang, Hai Helen Li, Yiran Chen

TL;DR

MUSE (Multimodal Unified Safety Evaluation), an open-source, run-centric platform that integrates automatic cross-modal payload generation, three multi-turn attack algorithms, provider-agnostic model routing, and an LLM judge with a five-level safety taxonomy into a single browser-based system is presented.

Abstract

Safety evaluation and red-teaming of large language models remain predominantly text-centric, and existing frameworks lack the infrastructure to systematically test whether alignment generalizes to audio, image, and video inputs. We present MUSE (Multimodal Unified Safety Evaluation), an open-source, run-centric platform that integrates automatic cross-modal payload generation, three multi-turn attack algorithms (Crescendo, PAIR, Violent Durian), provider-agnostic model routing, and an LLM judge with a five-level safety taxonomy into a single browser-based system. A dual-metric framework distinguishes hard Attack Success Rate (Compliance only) from soft ASR (including Partial Compliance), capturing partial information leakage that binary metrics miss. To probe whether alignment generalizes across modality boundaries, we introduce Inter-Turn Modality Switching (ITMS), which augments multi-turn attacks with per-turn modality rotation. Experiments across six multimodal LLMs from four providers show that multi-turn strategies can achieve up to 90-100% ASR against models with near-perfect single-turn refusal. ITMS does not uniformly raise final ASR on already-saturated baselines, but accelerates convergence by destabilizing early-turn defenses, and ablation reveals that the direction of modality effects is model-family-specific rather than universal, underscoring the need for provider-aware cross-modal safety testing.

MUSE: A Run-Centric Platform for Multimodal Unified Safety Evaluation of Large Language Models

TL;DR

MUSE (Multimodal Unified Safety Evaluation), an open-source, run-centric platform that integrates automatic cross-modal payload generation, three multi-turn attack algorithms, provider-agnostic model routing, and an LLM judge with a five-level safety taxonomy into a single browser-based system is presented.

Abstract

Safety evaluation and red-teaming of large language models remain predominantly text-centric, and existing frameworks lack the infrastructure to systematically test whether alignment generalizes to audio, image, and video inputs. We present MUSE (Multimodal Unified Safety Evaluation), an open-source, run-centric platform that integrates automatic cross-modal payload generation, three multi-turn attack algorithms (Crescendo, PAIR, Violent Durian), provider-agnostic model routing, and an LLM judge with a five-level safety taxonomy into a single browser-based system. A dual-metric framework distinguishes hard Attack Success Rate (Compliance only) from soft ASR (including Partial Compliance), capturing partial information leakage that binary metrics miss. To probe whether alignment generalizes across modality boundaries, we introduce Inter-Turn Modality Switching (ITMS), which augments multi-turn attacks with per-turn modality rotation. Experiments across six multimodal LLMs from four providers show that multi-turn strategies can achieve up to 90-100% ASR against models with near-perfect single-turn refusal. ITMS does not uniformly raise final ASR on already-saturated baselines, but accelerates convergence by destabilizing early-turn defenses, and ablation reveals that the direction of modality effects is model-family-specific rather than universal, underscoring the need for provider-aware cross-modal safety testing.
Paper Structure (23 sections, 4 figures, 4 tables)

This paper contains 23 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. System Design
  4. Overview
  5. Run-Centric Architecture
  6. Attack Strategies and ITMS
  7. Modality Conversion and Model Routing
  8. Evaluation Framework
  9. Experiments
  10. Setup
  11. Dataset.
  12. Models.
  13. Strategies and hyperparameters.
  14. Metrics.
  15. Single-Turn Baseline
  16. ...and 8 more sections

Figures (4)

  • Figure 1: MUSE system overview. The run-centric architecture connects cross-modal payload generation, multi-turn attack strategies, provider-agnostic model routing, and LLM-based safety judgment into a single browser-based platform.
  • Figure 2: Cumulative ASR (%) as a function of turn number, aggregated across all six target models per strategy. Markers at each turn; all five strategies share a 10-turn maximum budget.
  • Figure 3: Hard ASR (%) broken down by harm category and strategy, aggregated across all six target models. Categories (columns): Weapons (goals 0--9), Drugs (10--19), Malware (20--29), Bio/Eco (30--39), Fraud (40--49). Horizontal rule separates base strategies (top) from ITMS variants (bottom).
  • Figure 4: MUSE user interfaces.