Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unmasking the Canvas: A Dynamic Benchmark for Image Generation Jailbreaking and LLM Content Safety

Variath Madhupal Gautham Nair, Vishal Varma Dantuluri

TL;DR

The paper addresses the emergence of image-based jailbreaking in LLM-driven image generation and the lack of robust safety benchmarks. It introduces Unmasking the Canvas Benchmark (UTCB), a dynamic, scalable dataset comprising thousands of image-generation prompts with multilingual obfuscation, along with a modular evaluation pipeline and tiered annotation (Bronze/Silver/Gold). Using a Groq-hosted LLaMA-3 stack, zero-shot and fallback prompting, automated tagging, and an access-controlled annotation interface, the authors assess jailbreak susceptibility, risk scoring, and model responses, while highlighting misuse risks and defense opportunities. Key findings show scaleable jailbreak generation, language- and prompt-type dependencies in outcomes, and the defender-learner role of a Llama judge model, underscoring the need for evolving benchmarks to safeguard image-generation safety. Overall, UTCB provides a dynamic framework for ongoing defense against image-based jailbreaks, balancing public disclosure with ethical safeguards and community-driven data evolution.

Abstract

Existing large language models (LLMs) are advancing rapidly and produce outstanding results in image generation tasks, yet their content safety checks remain vulnerable to prompt-based jailbreaks. Through preliminary testing on platforms such as ChatGPT, MetaAI, and Grok, we observed that even short, natural prompts could lead to the generation of compromising images ranging from realistic depictions of forged documents to manipulated images of public figures. We introduce Unmasking the Canvas (UTC Benchmark; UTCB), a dynamic and scalable benchmark dataset to evaluate LLM vulnerability in image generation. Our methodology combines structured prompt engineering, multilingual obfuscation (e.g., Zulu, Gaelic, Base64), and evaluation using Groq-hosted LLaMA-3. The pipeline supports both zero-shot and fallback prompting strategies, risk scoring, and automated tagging. All generations are stored with rich metadata and curated into Bronze (non-verified), Silver (LLM-aided verification), and Gold (manually verified) tiers. UTCB is designed to evolve over time with new data sources, prompt templates, and model behaviors. Warning: This paper includes visual examples of adversarial inputs designed to test model safety. All outputs have been redacted to ensure responsible disclosure.

Unmasking the Canvas: A Dynamic Benchmark for Image Generation Jailbreaking and LLM Content Safety

TL;DR

The paper addresses the emergence of image-based jailbreaking in LLM-driven image generation and the lack of robust safety benchmarks. It introduces Unmasking the Canvas Benchmark (UTCB), a dynamic, scalable dataset comprising thousands of image-generation prompts with multilingual obfuscation, along with a modular evaluation pipeline and tiered annotation (Bronze/Silver/Gold). Using a Groq-hosted LLaMA-3 stack, zero-shot and fallback prompting, automated tagging, and an access-controlled annotation interface, the authors assess jailbreak susceptibility, risk scoring, and model responses, while highlighting misuse risks and defense opportunities. Key findings show scaleable jailbreak generation, language- and prompt-type dependencies in outcomes, and the defender-learner role of a Llama judge model, underscoring the need for evolving benchmarks to safeguard image-generation safety. Overall, UTCB provides a dynamic framework for ongoing defense against image-based jailbreaks, balancing public disclosure with ethical safeguards and community-driven data evolution.

Abstract

Existing large language models (LLMs) are advancing rapidly and produce outstanding results in image generation tasks, yet their content safety checks remain vulnerable to prompt-based jailbreaks. Through preliminary testing on platforms such as ChatGPT, MetaAI, and Grok, we observed that even short, natural prompts could lead to the generation of compromising images ranging from realistic depictions of forged documents to manipulated images of public figures. We introduce Unmasking the Canvas (UTC Benchmark; UTCB), a dynamic and scalable benchmark dataset to evaluate LLM vulnerability in image generation. Our methodology combines structured prompt engineering, multilingual obfuscation (e.g., Zulu, Gaelic, Base64), and evaluation using Groq-hosted LLaMA-3. The pipeline supports both zero-shot and fallback prompting strategies, risk scoring, and automated tagging. All generations are stored with rich metadata and curated into Bronze (non-verified), Silver (LLM-aided verification), and Gold (manually verified) tiers. UTCB is designed to evolve over time with new data sources, prompt templates, and model behaviors. Warning: This paper includes visual examples of adversarial inputs designed to test model safety. All outputs have been redacted to ensure responsible disclosure.
Paper Structure (21 sections, 10 figures)

This paper contains 21 sections, 10 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. JAILBREAKHUB.
  4. Competing Objectives, Mismatched Generalization and Multilingual Vulnerabilities.
  5. Tree of Attacks(TAP)
  6. Experimental Setup and Goals
  7. Large-scale prompt generation
  8. Llama as an Image Mimicker
  9. Unmasking the Canvas Prompt Interface
  10. UTCB Dataset
  11. Judge Model: Llama as the defender
  12. Evaluation and Results
  13. Large-scale Prompt Generation
  14. Large-scale Prompt Filtering
  15. Jailbreak analysis
  16. ...and 6 more sections

Figures (10)

  • Figure 1: Example of a successful image-based jailbreak on Grok model. The model responded with a policy-violating output. (Redacted)
  • Figure 2: Input prompt structure used to generate prompts at scale.
  • Figure 3: Number of prompts curated across types and languages.
  • Figure 4: Distribution of manually labeled v/s auto-tagged(using Llama) prompts, and prompts that are yet to be tested.
  • Figure 5: Number of prompts curated by the prompt generator model compared to manually curated prompts from authors.
  • ...and 5 more figures