Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming

TL;DR

The paper addresses the invalidity of many ASR-based comparisons in AI red teaming by applying social science measurement theory to jailbreaking. It reframes ASRs as measurements of well-specified estimands derived from a probabilistic threat model, introducing a two-part sufficiency condition: conceptual coherence and measurement validity. Through analysis and case studies, it reveals how mismatches in estimands (e.g., Top-1 vs one-shot) and measurement mis-specifications (e.g., D vs D, J vs s, differential judge errors) bias conclusions about safety and attack efficacy. The authors provide practical recommendations to improve validity and cross-study comparability, with implications for broader genAI evaluation beyond jailbreaking.

Abstract

We argue that conclusions drawn about relative system safety or attack method efficacy via AI red teaming are often not supported by evidence provided by attack success rate (ASR) comparisons. We show, through conceptual, theoretical, and empirical contributions, that many conclusions are founded on apples-to-oranges comparisons or low-validity measurements. Our arguments are grounded in asking a simple question: When can attack success rates be meaningfully compared? To answer this question, we draw on ideas from social science measurement theory and inferential statistics, which, taken together, provide a conceptual grounding for understanding when numerical values obtained through the quantification of system attributes can be meaningfully compared. Through this lens, we articulate conditions under which ASRs can and cannot be meaningfully compared. Using jailbreaking as a running example, we provide examples and extensive discussion of apples-to-oranges ASR comparisons and measurement validity challenges.

Figures (5)

• Figure 1: Jailbreaking instantiated in a formal measurement theory framework. The diagram shows how $\mathop{\mathrm{ASR}}\nolimits$s can be viewed as estimates (measurements) of precisely-defined estimands (attack success probabilities). The processes of systematization, operationalization and execution connect ASRs obtained in the "observed activity" to the concepts they are intended to measure. By separating operationalization from systematization, the measurement framework distinguishes between measurement error (the estimation error between the estimate $\mathop{\mathrm{ASR}}\nolimits$ and the target estimand, $\alpha$) and conceptual gaps (deficiencies in how a concept such as safety is systematized).
• Figure 2: Effect of repeated sampling (Top-1 aggregation) and decoding configuration on $\mathop{\mathrm{ASR}}\nolimits$ using 100 prompts from MaliciousInstruct. A and B: Top-1 $\mathop{\mathrm{ASR}}\nolimits$ using "attacker scorer" to select top response (grey) for success adjudication by $J$ vs. using $J$ to both select the top response and adjudicate success (orange). One-shot $\mathop{\mathrm{ASR}}\nolimits$ shown in green. C: Histograms of estimated per-prompt one-shot success probabilities. While the average one-shot $\mathop{\mathrm{ASR}}\nolimits$ (green curve from A) is mostly flat except at the highest temps, entropy of per-prompt success probability distribution increases greatly with temperature.
• Figure 3: Figures illustrate the bias introduced by judge error by showing how much $\mathbb{E}(\mathop{\mathrm{ASR}}\nolimits) = \mathbb{P}(J(L(P)) = 1)$ can deviate from the estimand, $\alpha$, depending on the judge system TPR and FPR with respect to the true success criterion $s$. Left: Concrete example of differential judge error introducing bias into cross-system safety comparison. Middle: Relationship between $\mathbb{E}(\mathop{\mathrm{ASR}}\nolimits)$ and $\alpha$ as TPR and FPR vary along an ROC curve with AUC $0.91$. Right: Relationship between $\mathbb{E}(\mathop{\mathrm{ASR}}\nolimits)$ and $\alpha$ as TPR and FPR vary subject to holding judge accuracy $\mathbb{P}(J = s)$ fixed at $0.7, 0.8, 0.9$.
• Figure 4: Experiments on 160 prompts from chu2024comprehensive. We use the same judge as for our GE experiments, which is not the same as the judge used in chu2024comprehensive, so values here should be compared to the original paper with caution. Just as in our replication of the GE experiments using the 100 prompts from MaliciousInstruct (Figure \ref{['app:fig:repeated_sampling']}), we find that one-shot $\mathop{\mathrm{ASR}}\nolimits$ (green) does not change much across configurations. But we see clearly that as temperature increases, the entropy of the per-prompt attack success probability distribution greatly increases. In particular, fewer base prompts have a statistically $0$% chance of producing undesirable responses. Applying Top-1 aggregation over repeated sampling in such settings produces very high observed $\mathop{\mathrm{ASR}}\nolimits$s.
• Figure 5: In the main paper we presented results for the Llama 2 13B Chat model. Shown here are $\mathop{\mathrm{ASR}}\nolimits$ vs. configuration results for the other 3 models we conducted experiments on. The base Llama models do not undergo the same safety alignment so it is most interesting to consider $\mathop{\mathrm{ASR}}\nolimits$s for the two Chat variants. Llama 2 7B Chat shows a more significant upward trend in one-shot $\mathop{\mathrm{ASR}}\nolimits$ (green curve) as temperature increases than what is observed for Llama 2 13B Chat.

Theorems & Definitions (1)

• Definition 3.1: Probabilistic threat model for jailbreaking