Diversifying Toxicity Search in Large Language Models Through Speciation

TL;DR

The paper tackles the limitation of red-teaming LLM toxicity with a single dominant prompt by introducing ToxSearch-S, a Quality-Diversity based system that maintains multiple, behaviorally distinct species of toxic prompts. It uses online speciation via Leader-Follower clustering on an ensemble distance that combines semantic and behavioral similarity, formalized as $d_{ensemble}(u,v) = w_{genotype} \cdot d_{genotype\_norm}(u,v) + w_{phenotype} \cdot d_{phenotype}(u,v)$ with $w_{genotype}=0.7$, $w_{phenotype}=0.3$, $d_{genotype\_norm}=d_{genotype}/2$, $d_{genotype}=1 - e_u^T e_v$, and $d_{phenotype}(u,v)=\frac{\|s(y_u) - s(y_v)\|_2}{\sqrt{8}}$, where $s(y) \in [0,1]^8$. The framework defines a QD objective $\max_{\{S_1, \dots, S_k\}} \sum_{i=1}^k \max_{p \in S_i} \hat{f}(p)$ subject to $D_{inter}(\{S_i\}) \geq \theta_{diversity}$ to preserve diverse, high-toxicity niches. Empirically, ToxSearch-S achieves higher peak toxicity ($\approx 0.73$) and a heavier extreme tail (top-10 median $\approx 0.66$) than the baseline, while expanding semantic coverage (higher $N_1$ and $K$) and maintaining well-separated species with distinct toxicity distributions (mean separation ratio $\approx 1.93$). These results demonstrate that explicit speciation illuminates multiple attack strategies, enabling broader, parallel red-teaming coverage and more robust safety evaluation.

Abstract

Evolutionary prompt search is a practical black-box approach for red teaming large language models (LLMs), but existing methods often collapse onto a small family of high-performing prompts, limiting coverage of distinct failure modes. We present a speciated quality-diversity (QD) extension of ToxSearch that maintains multiple high-toxicity prompt niches in parallel rather than optimizing a single best prompt. ToxSearch-S introduces unsupervised prompt speciation via a search methodology that maintains capacity-limited species with exemplar leaders, a reserve pool for outliers and emerging niches, and species-aware parent selection that trades off within-niche exploitation and cross-niche exploration. ToxSearch-S is found to reach higher peak toxicity ($\approx 0.73$ vs.\ $\approx 0.47$) and a extreme heavier tail (top-10 median $0.66$ vs.\ $0.45$) than the baseline, while maintaining comparable performance on moderately toxic prompts. Speciation also yields broader semantic coverage under a topic-as-species analysis (higher effective topic diversity $N_1$ and larger unique topic coverage $K$). Finally, species formed are well-separated in embedding space (mean separation ratio $\approx 1.93$) and exhibit distinct toxicity distributions, indicating that speciation partitions the adversarial space into behaviorally differentiated niches rather than superficial lexical variants. This suggests our approach uncovers a wider range of attack strategies.

Figures (7)

• Figure 1: Cumulative maximum toxicity (solid) and Avg fitness (dashed) over generations
• Figure 2: ECDF of prompt toxicities with markers for $Q_{0.95}$, top-10 median, and $Q_{\max}$. Speciated ToxSearch discovers more extreme toxic prompts (top-10 median: 0.66 vs 0.45).
• Figure 3: Topic diversity comparison: (A) effective number of topics $N_1 = \exp(H)$, (B) unique topic count $K$. Speciated maintains significantly increased semantic diversity.
• Figure 4: MDS visualization of prompt embeddings. Color indicates condition (blue: baseline, red: speciated); shade intensity reflects max toxicity per topic.
• Figure 5: Toxicity Distribution by Species. Horizontal boxplots with jittered strip plots show the distribution of toxicity scores for the top 10 species groups (grouped by exact 10-label match across runs). Species are ordered by max toxicity (descending) and colored where top-performing species receive darker, more vibrant colors. The boxplots reveal distinct toxicity distributions for each species, demonstrating that different species occupy different regions of the toxicity space.