Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TokenProber: Jailbreaking Text-to-image Models via Fine-grained Word Impact Analysis

Longtian Wang, Xiaofei Xie, Tianlin Li, Yuhan Zhi, Chao Shen

TL;DR

TokenProber addresses the risk of NSFW content slipping through safety mechanisms in text-to-image models by performing sensitivity-aware, word-level analysis to craft adversarial prompts. It introduces two word types—dirty words essential for NSFW generation and discrepant words that shift safety checker decisions—and combines dirtiness-preserving and discrepancy-away mutations within a differential-testing framework guided by a surrogate safety checker. Empirical results across multiple T2I models and safety checkers show TokenProber yields a 54%+ bypass-rate improvement on average and requires substantially less time than prior methods. The work highlights gaps in current safety filters, suggests ensemble-filtering strategies, and provides a generalizable paradigm for robustness testing of safety mechanisms in generative systems.

Abstract

Text-to-image (T2I) models have significantly advanced in producing high-quality images. However, such models have the ability to generate images containing not-safe-for-work (NSFW) content, such as pornography, violence, political content, and discrimination. To mitigate the risk of generating NSFW content, refusal mechanisms, i.e., safety checkers, have been developed to check potential NSFW content. Adversarial prompting techniques have been developed to evaluate the robustness of the refusal mechanisms. The key challenge remains to subtly modify the prompt in a way that preserves its sensitive nature while bypassing the refusal mechanisms. In this paper, we introduce TokenProber, a method designed for sensitivity-aware differential testing, aimed at evaluating the robustness of the refusal mechanisms in T2I models by generating adversarial prompts. Our approach is based on the key observation that adversarial prompts often succeed by exploiting discrepancies in how T2I models and safety checkers interpret sensitive content. Thus, we conduct a fine-grained analysis of the impact of specific words within prompts, distinguishing between dirty words that are essential for NSFW content generation and discrepant words that highlight the different sensitivity assessments between T2I models and safety checkers. Through the sensitivity-aware mutation, TokenProber generates adversarial prompts, striking a balance between maintaining NSFW content generation and evading detection. Our evaluation of TokenProber against 5 safety checkers on 3 popular T2I models, using 324 NSFW prompts, demonstrates its superior effectiveness in bypassing safety filters compared to existing methods (e.g., 54%+ increase on average), highlighting TokenProber's ability to uncover robustness issues in the existing refusal mechanisms.

TokenProber: Jailbreaking Text-to-image Models via Fine-grained Word Impact Analysis

TL;DR

TokenProber addresses the risk of NSFW content slipping through safety mechanisms in text-to-image models by performing sensitivity-aware, word-level analysis to craft adversarial prompts. It introduces two word types—dirty words essential for NSFW generation and discrepant words that shift safety checker decisions—and combines dirtiness-preserving and discrepancy-away mutations within a differential-testing framework guided by a surrogate safety checker. Empirical results across multiple T2I models and safety checkers show TokenProber yields a 54%+ bypass-rate improvement on average and requires substantially less time than prior methods. The work highlights gaps in current safety filters, suggests ensemble-filtering strategies, and provides a generalizable paradigm for robustness testing of safety mechanisms in generative systems.

Abstract

Text-to-image (T2I) models have significantly advanced in producing high-quality images. However, such models have the ability to generate images containing not-safe-for-work (NSFW) content, such as pornography, violence, political content, and discrimination. To mitigate the risk of generating NSFW content, refusal mechanisms, i.e., safety checkers, have been developed to check potential NSFW content. Adversarial prompting techniques have been developed to evaluate the robustness of the refusal mechanisms. The key challenge remains to subtly modify the prompt in a way that preserves its sensitive nature while bypassing the refusal mechanisms. In this paper, we introduce TokenProber, a method designed for sensitivity-aware differential testing, aimed at evaluating the robustness of the refusal mechanisms in T2I models by generating adversarial prompts. Our approach is based on the key observation that adversarial prompts often succeed by exploiting discrepancies in how T2I models and safety checkers interpret sensitive content. Thus, we conduct a fine-grained analysis of the impact of specific words within prompts, distinguishing between dirty words that are essential for NSFW content generation and discrepant words that highlight the different sensitivity assessments between T2I models and safety checkers. Through the sensitivity-aware mutation, TokenProber generates adversarial prompts, striking a balance between maintaining NSFW content generation and evading detection. Our evaluation of TokenProber against 5 safety checkers on 3 popular T2I models, using 324 NSFW prompts, demonstrates its superior effectiveness in bypassing safety filters compared to existing methods (e.g., 54%+ increase on average), highlighting TokenProber's ability to uncover robustness issues in the existing refusal mechanisms.
Paper Structure (38 sections, 5 equations, 5 figures, 4 tables, 1 algorithm)

This paper contains 38 sections, 5 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Text-to-image Model
  4. Safety checker for T2I Model
  5. Problem Definition
  6. Methodology
  7. Overview of our Approach
  8. Word-level Sensitivity Analysis
  9. Dirtiness Analysis
  10. Discrepancy Analysis
  11. Sensitivity-aware Differential Testing
  12. Dirtiness-Preserving Mutation
  13. Discrepancy-Away Mutation
  14. Oracle and Fitness
  15. Discussion
  16. ...and 23 more sections

Figures (5)

  • Figure 1: Influence of different words within prompts. The explicit use of dirty words (e.g., 'naked') can lead T2I models to generate NSFW content, which simultaneously triggers detection by the safety checkers. However, due to differing decision boundaries between the T2I model and the safety checkers regarding NSFW content, some words (e.g., 'diddling') may fall within the region where discrepancies exist, specifically in the overlap between the boundaries of the safety checker and the T2I model. These words can cause the T2I model to generate NSFW content without being detected by the safety checker.
  • Figure 2: Text-to-image model structure
  • Figure 3: The workflow of TokenProber
  • Figure 4: Results of TokenProber configured with different parameters
  • Figure 5: Note: Figure contains profanity. The impact of dirty words and discrepant words

Theorems & Definitions (1)

  • Definition 1: Robustness of Safety Checker for T2I Model