Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation

TL;DR

This work tackles the problem of safety in text-to-image generation by proposing Adversarial Nibbler, a public red-teaming framework that crowdsources implicitly adversarial prompts to uncover long-tail harms. It combines a data-centric evaluation approach with a multi-model T2I suite, human harm annotations, and an interactive visualization tool to enable continuous auditing of safety failures. Round 1 produced a large, richly annotated dataset (over 10k prompt-image pairs with machine labels and ~1.5k with human harm annotations), revealed substantial gaps in automated safety classifications, and uncovered novel attack strategies. The study demonstrates the value of open, crowdsourced safety benchmarking for proactive, iterative risk assessment and provides practical recommendations for red-teaming practices and benchmarking in T2I systems.

Abstract

With the rise of text-to-image (T2I) generative AI models reaching wide audiences, it is critical to evaluate model robustness against non-obvious attacks to mitigate the generation of offensive images. By focusing on ``implicitly adversarial'' prompts (those that trigger T2I models to generate unsafe images for non-obvious reasons), we isolate a set of difficult safety issues that human creativity is well-suited to uncover. To this end, we built the Adversarial Nibbler Challenge, a red-teaming methodology for crowdsourcing a diverse set of implicitly adversarial prompts. We have assembled a suite of state-of-the-art T2I models, employed a simple user interface to identify and annotate harms, and engaged diverse populations to capture long-tail safety issues that may be overlooked in standard testing. The challenge is run in consecutive rounds to enable a sustained discovery and analysis of safety pitfalls in T2I models. In this paper, we present an in-depth account of our methodology, a systematic study of novel attack strategies and discussion of safety failures revealed by challenge participants. We also release a companion visualization tool for easy exploration and derivation of insights from the dataset. The first challenge round resulted in over 10k prompt-image pairs with machine annotations for safety. A subset of 1.5k samples contains rich human annotations of harm types and attack styles. We find that 14% of images that humans consider harmful are mislabeled as ``safe'' by machines. We have identified new attack strategies that highlight the complexity of ensuring T2I model robustness. Our findings emphasize the necessity of continual auditing and adaptation as new vulnerabilities emerge. We are confident that this work will enable proactive, iterative safety assessments and promote responsible development of T2I models.

Figures (6)

• Figure 1: These images were generated by state-of-the-art models in response to textual prompts. In order to avoid displaying potentially objectionable visual content to readers, we have intentionally blurred the images.
• Figure 2: The Adversarial Nibbler User Journey. [Step 1] Participant inputs prompt into the platform. [Step 2] The model generates up to 12 images from 5 different T2I models and the user selects a harmful image [Step 3] The user answers 4 questions about the prompt and the image, and clicks the "Submit" button to record their discovery.
• Figure 3: A list of the questions used to annotate prompt-image pairs submitted to the Nibbler Challenge
• Figure 4: Heatmaps of the top-20 unigrams in the different failure categories
• Figure 5: The first two pages of the validation interface, showing the annotation questions listed with each prompt-image pair. Wording for the "harms," "identity attributes," and "attack modes" is kept consistent between the submission interface an the validation interface.