Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

Sunishchal Dev, Andrew Sloan, Joshua Kavner, Nicholas Kong, Morgan Sandler

TL;DR

The Judge Reliability Harness is presented, an open source library for constructing validation suites that test the reliability of LLM judges and finds meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges.

Abstract

We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

TL;DR

The Judge Reliability Harness is presented, an open source library for constructing validation suites that test the reliability of LLM judges and finds meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges.

Abstract

We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness
Paper Structure (23 sections, 4 figures, 6 tables)

This paper contains 23 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Basic Perturbations
  5. Stochastic Stability
  6. Synthetic Ordinal
  7. Agentic Mode
  8. Human in the Loop Review
  9. Experiments
  10. Benchmarks
  11. Human in the Loop Validation
  12. Evaluation Protocol
  13. LLM Judges
  14. Results
  15. Discussion
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Workflow for the Judge Reliability Harness. Human-in-the-loop review is performed after generation of the JRH synthetic test suite and before evaluation by the judges.
  • Figure 2: Human in the Loop Review User Interface (Agentic mode). The original sample from the benchmark is shown at the top. The left shows the transcript provided as input along with edits made by the synthetic data pipeline in red and green. The right shows the edited transcript, which is a free text box the user may use to make further edits. The bottom gives the user an option to accept the sample if they are satisfied with the edits, or to reject the sample and move on to the next one.
  • Figure 3: Reliability performance heatmaps across all four benchmarks: (a) FORTRESS, (b) HarmBench, (c) Persuade, and (d) AgentHarm.
  • Figure 4: Cost-Accuracy Trade-off Across Benchmarks. Each point represents a model's total cost and average accuracy across all benchmarks. Full model names are Llama Maverick 4, Gemini 2.5 Pro, GPT-4o, and Claude Sonnet 4.5