Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA

Xanh Ho, Jiahao Huang, Florian Boudin, Akiko Aizawa

TL;DR

The paper challenges the sufficiency of Exact Match and F1 metrics for evaluating extractive QA by introducing LLM-as-a-judge as an alternative. It conducts a multi-dataset study (Quoref, DROP, HotpotQA, 2Wiki) across diverse answer types and three judge-model families, comparing judge scores to human judgments. The results show a strong alignment between LLM-as-a-judge scores and human judgments (approximately 0.85 Pearson correlation), significantly higher than EM/F1 correlations, suggesting EM/F1 underestimates true model performance. The analysis reveals nuanced insights by answer type and finds minimal self-bias when the same model is used for both QA and judging, while also highlighting that some types (e.g., jobs) pose challenges. The work advocates adopting LLM-as-a-judge as a primary evaluation tool and provides a dataset and prompting strategies to facilitate broader adoption.

Abstract

Extractive reading comprehension question answering (QA) datasets are typically evaluated using Exact Match (EM) and F1-score, but these metrics often fail to fully capture model performance. With the success of large language models (LLMs), they have been employed in various tasks, including serving as judges (LLM-as-a-judge). In this paper, we reassess the performance of QA models using LLM-as-a-judge across four reading comprehension QA datasets. We examine different families of LLMs and various answer types to evaluate the effectiveness of LLM-as-a-judge in these tasks. Our results show that LLM-as-a-judge is highly correlated with human judgments and can replace traditional EM/F1 metrics. By using LLM-as-a-judge, the correlation with human judgments improves significantly, from 0.22 (EM) and 0.40 (F1-score) to 0.85. These findings confirm that EM and F1 metrics underestimate the true performance of the QA models. While LLM-as-a-judge is not perfect for more difficult answer types (e.g., job), it still outperforms EM/F1, and we observe no bias issues, such as self-preference, when the same model is used for both the QA and judgment tasks.

LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA

TL;DR

The paper challenges the sufficiency of Exact Match and F1 metrics for evaluating extractive QA by introducing LLM-as-a-judge as an alternative. It conducts a multi-dataset study (Quoref, DROP, HotpotQA, 2Wiki) across diverse answer types and three judge-model families, comparing judge scores to human judgments. The results show a strong alignment between LLM-as-a-judge scores and human judgments (approximately 0.85 Pearson correlation), significantly higher than EM/F1 correlations, suggesting EM/F1 underestimates true model performance. The analysis reveals nuanced insights by answer type and finds minimal self-bias when the same model is used for both QA and judging, while also highlighting that some types (e.g., jobs) pose challenges. The work advocates adopting LLM-as-a-judge as a primary evaluation tool and provides a dataset and prompting strategies to facilitate broader adoption.

Abstract

Extractive reading comprehension question answering (QA) datasets are typically evaluated using Exact Match (EM) and F1-score, but these metrics often fail to fully capture model performance. With the success of large language models (LLMs), they have been employed in various tasks, including serving as judges (LLM-as-a-judge). In this paper, we reassess the performance of QA models using LLM-as-a-judge across four reading comprehension QA datasets. We examine different families of LLMs and various answer types to evaluate the effectiveness of LLM-as-a-judge in these tasks. Our results show that LLM-as-a-judge is highly correlated with human judgments and can replace traditional EM/F1 metrics. By using LLM-as-a-judge, the correlation with human judgments improves significantly, from 0.22 (EM) and 0.40 (F1-score) to 0.85. These findings confirm that EM and F1 metrics underestimate the true performance of the QA models. While LLM-as-a-judge is not perfect for more difficult answer types (e.g., job), it still outperforms EM/F1, and we observe no bias issues, such as self-preference, when the same model is used for both the QA and judgment tasks.

Paper Structure

This paper contains 33 sections, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Extractive QA.
  4. Evaluation of Extractive QA.
  5. LLM-as-a-Judge.
  6. Experimental Settings
  7. Datasets
  8. Quoref.
  9. DROP.
  10. HotpotQA.
  11. 2Wiki.
  12. Obtaining Answer Types.
  13. Models
  14. QA Task.
  15. LLM-as-a-Judge.
  16. ...and 18 more sections

Figures (3)

  • Figure 1: An example that shows EM and F1-score underestimate the performance of models, while LLM-as-a-judge provides a more robust perspective.
  • Figure 2: Correlation scores of Qwen-as-a-judge with human judgment for each answer type.
  • Figure 3: Percentage of each answer type identified as correct by Qwen-as-a-judge.