ExtremeAIGC: Benchmarking LMM Vulnerability to AI-Generated Extremist Content

TL;DR

ExtremeAIGC introduces a dedicated benchmark to assess LMM safety against AI-generated extremist content by combining AI-generated imagery with real-world events. The framework evaluates four jailbreaking techniques across six state-of-the-art LMMs using generation-based and optimization-based attacks, revealing significant safety gaps. The dataset comprises 3,141 high-quality images from 1,047 prompts across 29 extremist events, enabling robust testing of cross-modal defenses and attack transfer. Findings demonstrate substantial vulnerability of current LMM safety mechanisms, underscoring the need for more robust, adaptive, and cross-modal mitigation strategies with real-world safety implications.

Abstract

Large Multimodal Models (LMMs) are increasingly vulnerable to AI-generated extremist content, including photorealistic images and text, which can be used to bypass safety mechanisms and generate harmful outputs. However, existing datasets for evaluating LMM robustness offer limited exploration of extremist content, often lacking AI-generated images, diverse image generation models, and comprehensive coverage of historical events, which hinders a complete assessment of model vulnerabilities. To fill this gap, we introduce ExtremeAIGC, a benchmark dataset and evaluation framework designed to assess LMM vulnerabilities against such content. ExtremeAIGC simulates real-world events and malicious use cases by curating diverse text- and image-based examples crafted using state-of-the-art image generation techniques. Our study reveals alarming weaknesses in LMMs, demonstrating that even cutting-edge safety measures fail to prevent the generation of extremist material. We systematically quantify the success rates of various attack strategies, exposing critical gaps in current defenses and emphasizing the need for more robust mitigation strategies.

Figures (7)

• Figure 1: Impact of multimodal inputs (text and image) and jailbreaking on generative model responses. The graph reveals a significant surge in LMM failures when subjected to jailbreaking attacks.
• Figure 2: Dataset Statistics. a) shows the distribution of our 29 historical events across the time range of 1822 to 2024, b) shows the distribution of 91 event attributes across time, c) shows the distribution of images across different topics.
• Figure 3: Overview of the experimental setup for evaluating multimodal model vulnerabilities using four jailbreaking methods. The setup includes two generation-based and two optimization-based methods. The adversarial inputs are fed into five SOTA multimodal models, and their responses are analyzed based on Attack Success Rate (ASR).
• Figure 4: Heatmaps indicating vulnerable regions in the LLAVA model for three different attack scenarios.
• Figure 5: Jailbreaking Experiment on a sample AI-generated image for all 4 types. As we can observe, all 4 were able to bypass for the example image using MiniGPT4 model. It also covers all types of topics.