Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges

TL;DR

This work interrogates the reliability of LLM-based safety judges used for offline benchmarking and online guardrails by evaluating robustness to real-world distribution shifts and adversarial attacks. Through meta-evaluations on four open-source judges and a JailbreakBench-derived dataset, the study reveals that stylistic alterations to outputs and adversarial output-level modifications can meaningfully worsen judgment quality, with some attacks yielding $FNR$ as high as $1.0$ and others causing notable drops in $F1$. HarmBench generally shows greater resilience to adversarial attacks, but all judges exhibit notable vulnerabilities, especially under storytelling styles that raise misclassification rates. The findings argue for comprehensive threat modeling and more robust, domain-specific safety evaluators to avoid a false sense of security in deployed AI systems.

Abstract

Large Language Model (LLM) based judges form the underpinnings of key safety evaluation processes such as offline benchmarking, automated red-teaming, and online guardrailing. This widespread requirement raises the crucial question: can we trust the evaluations of these evaluators? In this paper, we highlight two critical challenges that are typically overlooked: (i) evaluations in the wild where factors like prompt sensitivity and distribution shifts can affect performance and (ii) adversarial attacks that target the judge. We highlight the importance of these through a study of commonly used safety judges, showing that small changes such as the style of the model output can lead to jumps of up to 0.24 in the false negative rate on the same dataset, whereas adversarial attacks on the model generation can fool some judges into misclassifying 100% of harmful generations as safe ones. These findings reveal gaps in commonly used meta-evaluation benchmarks and weaknesses in the robustness of current LLM judges, indicating that low attack success under certain judges could create a false sense of security.

Figures (4)

• Figure 1: Stylistic Prompt Formatting: given a seed dataset of model responses, a re-styling model creates a new dataset with the same (or very similar) harmfulness labels but different generations.
• Figure 2: Output-level Modifications: output-level modifications simulate an adversary (either through a malicious model or adversarial input) that manages to add additional instructions to the generated model output that specifically target the judge.
• Figure 3: Judge Confusion Matrices: under (a) the baseline, and (b) the "Prepend + Append Benign" output modification.
• Figure 4: "Append Long Benign" Ablation: effect of length of the benign prompt on FNR and F1 for HarmBench.