OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs

TL;DR

OpenRT tackles the lack of scalable, systematic safety evaluation for multimodal LLMs by delivering a modular, open-source red-teaming framework with an adversarial kernel that decouples model interfaces, datasets, attacks, judges, and evaluators. It supports 37 attack methods across white-box and black-box settings and employs a high-throughput asynchronous runtime to test 20 advanced MLLMs, revealing widespread safety vulnerabilities with an average ASR of $49.14\%$ and extreme variability across attack paradigms. Key contributions include a comprehensive modular registry, a dual-judge evaluation system, and extensive empirical insights showing that frontier models remain susceptible under adaptive, multi-turn, and multimodal jailbreaks, highlighting the need for defense-in-depth and continuous red-teaming. By open-sourcing OpenRT, the work establishes a sustainable infrastructure to standardize and accelerate safety improvements across models and deployment contexts.

Abstract

The rapid integration of Multimodal Large Language Models (MLLMs) into critical applications is increasingly hindered by persistent safety vulnerabilities. However, existing red-teaming benchmarks are often fragmented, limited to single-turn text interactions, and lack the scalability required for systematic evaluation. To address this, we introduce OpenRT, a unified, modular, and high-throughput red-teaming framework designed for comprehensive MLLM safety evaluation. At its core, OpenRT architects a paradigm shift in automated red-teaming by introducing an adversarial kernel that enables modular separation across five critical dimensions: model integration, dataset management, attack strategies, judging methods, and evaluation metrics. By standardizing attack interfaces, it decouples adversarial logic from a high-throughput asynchronous runtime, enabling systematic scaling across diverse models. Our framework integrates 37 diverse attack methodologies, spanning white-box gradients, multi-modal perturbations, and sophisticated multi-agent evolutionary strategies. Through an extensive empirical study on 20 advanced models (including GPT-5.2, Claude 4.5, and Gemini 3 Pro), we expose critical safety gaps: even frontier models fail to generalize across attack paradigms, with leading models exhibiting average Attack Success Rates as high as 49.14%. Notably, our findings reveal that reasoning models do not inherently possess superior robustness against complex, multi-turn jailbreaks. By open-sourcing OpenRT, we provide a sustainable, extensible, and continuously maintained infrastructure that accelerates the development and standardization of AI safety.

Figures (11)

• Figure 1: Left: Average Attack Success Rate (ASR) of OpenRT across MLLMs. Right: Comparison against the top-6 strongest attack baselines (highest ASR) on representative MLLMs (underlined in legend).
• Figure 2: Computational cost of various attack methods against GPT-5.2, measured by input and output token usage (log$_{10}$ scale).
• Figure 3: Attack stealthiness against GPT-5.2 measured by perplexity (PPL) on Qwen3-32B (log$_{10}$ scale). Lower PPL indicates more stealthy prompts. Shaded bands mark high ($<10$), moderate ($10$--$30$), and low ($>30$) stealthiness.
• Figure 4: Diversity analysis of attack methods against GPT-5.2, quantified by the mean pairwise cosine distance between embeddings of successful adversarial prompts.
• Figure : Installing OpenRT from source.