RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following

Abstract

Rubric-based evaluation has become a prevailing paradigm for evaluating instruction following in large language models (LLMs). Despite its widespread use, the reliability of these rubric-level evaluations remains unclear, calling for meta-evaluation. However, prior meta-evaluation efforts largely focus on the response level, failing to assess the fine-grained judgment accuracy that rubric-based evaluation relies on. To bridge this gap, we introduce RubricEval. Our benchmark features: (1) the first rubric-level meta-evaluation benchmark for instruction following, (2) diverse instructions and responses spanning multiple categories and model sources, and (3) a substantial set of 3,486 quality-controlled instances, along with Easy/Hard subsets that better differentiates judge performance. Our experiments reveal that rubric-level judging remains far from solved: even GPT-4o, a widely adopted judge in instruction-following benchmarks, achieves only 55.97% on Hard subset. Considering evaluation paradigm, rubric-level evaluation outperforms checklist-level, explicit reasoning improves accuracy, and both together reduce inter-judge variance. Through our established rubric taxonomy, we further identify common failure modes and offer actionable insights for reliable instruction-following evaluation.

Figures (9)

• Figure 1: Existing rubric-based instruction-following evaluation and our rubric-level meta-evaluation task.
• Figure 2: Overview of our data construction pipeline.
• Figure 3: Preliminary experiments and results on human reference set.
• Figure 4: Distribution of instances across rubric categories in our taxonomy.
• Figure 5: Inter-judge analysis on CFBench. Judge variance decreases from vanilla (*) to rubric-level with reasoning.