Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions

TL;DR

The paper investigates how prompting Granularity affects LLMs-as-a-judge's alignment with human judgments across NLG tasks. It introduces a four-category quality-criteria taxonomy and evaluates eight benchmarks with multiple LLM families under four prompting settings, plus a perplexity-based baseline. Key findings show that detailed rubrics yield only modest improvements for large models, while perplexity often matches or exceeds prompting performance for textual quality. Larger models remain necessary for complex, task-specific judgments such as relevance and logical integrity, informing practical design of automatic evaluation pipelines.

Abstract

LLMs-as-a-judge is a recently popularized method which replaces human judgements in task evaluation (Zheng et al. 2024) with automatic evaluation using LLMs. Due to widespread use of RLHF (Reinforcement Learning from Human Feedback), state-of-the-art LLMs like GPT4 and Llama3 are expected to have strong alignment with human preferences when prompted for a quality judgement, such as the coherence of a text. While this seems beneficial, it is not clear whether the assessments by an LLM-as-a-judge constitute only an evaluation based on the instructions in the prompts, or reflect its preference for high-quality data similar to its fine-tune data. To investigate how much influence prompting the LLMs-as-a-judge has on the alignment of AI judgements to human judgements, we analyze prompts with increasing levels of instructions about the target quality of an evaluation, for several LLMs-as-a-judge. Further, we compare to a prompt-free method using model perplexity as a quality measure instead. We aggregate a taxonomy of quality criteria commonly used across state-of-the-art evaluations with LLMs and provide this as a rigorous benchmark of models as judges. Overall, we show that the LLMs-as-a-judge benefit only little from highly detailed instructions in prompts and that perplexity can sometimes align better with human judgements than prompting, especially on textual quality.

Figures (6)

• Figure 1: Our prompting settings. We measure how much influence the information about the actual evaluation has for model performance as LLM-as-a-judge. For setting 1, perplexity, we don't prompt the models but calculate the models' perplexity for the task solution in the example instead. The example prompts shown above are used for the LLMs-as-a-judge to measure the quality for the criterion logicality as defined in for the benchmark dataset TheNextChapter.
• Figure 2: Taxonomy of quality criteria summarizing current state-of-the-art benchmark datasets and criteria used for automatic evaluations with LLMs. We group all 34 quality criteria as defined in the 8 different benchmark datasets into 4 groups: Content-based, Engagement-based, Integrity-based, Relevance-based criteria.
• Figure 3: Radar chart of average Pearson correlations for the quality criteria groups for each of different settings of prompting (1 - Perplexity / No Prompting, 2 - Generic prompt, 3 - Specific prompt, 4 - Full rubric) over all models.
• Figure 4: Radar chart of average Pearson correlations for the quality criteria groups for each of the different LLMs-as-a-judge over all setting of prompting.
• Figure 5: Violin plot of the generated scores by the LLMs-as-a-judge for the engagement-based criterion empathy, together with the corresponding human annotations.