Reference-based Metrics Disprove Themselves in Question Generation

TL;DR

A reference-free metric consisted of multi-dimensional criteria such as naturalness, answerability, and complexity, utilizing large language models is proposed, and it is revealed that this metric accurately distinguishes between high-quality questions and flawed ones, and achieves state-of-the-art alignment with human judgment.

Abstract

Reference-based metrics such as BLEU and BERTScore are widely used to evaluate question generation (QG). In this study, on QG benchmarks such as SQuAD and HotpotQA, we find that using human-written references cannot guarantee the effectiveness of the reference-based metrics. Most QG benchmarks have only one reference; we replicate the annotation process and collect another reference. A good metric is expected to grade a human-validated question no worse than generated questions. However, the results of reference-based metrics on our newly collected reference disproved the metrics themselves. We propose a reference-free metric consisted of multi-dimensional criteria such as naturalness, answerability, and complexity, utilizing large language models. These criteria are not constrained to the syntactic or semantic of a single reference question, and the metric does not require a diverse set of references. Experiments reveal that our metric accurately distinguishes between high-quality questions and flawed ones, and achieves state-of-the-art alignment with human judgment.

Figures (6)

• Figure 1: Normalized value of different evaluation metrics for four types of candidate questions against the same reference (RefQ) in the HotpotQA dataset yang2018hotpotqa. Ideally, metrics should score Group 1 highest. Current QG metrics, except for NACo (ours) and RQUGE, primarily recognize random questions (Group 4) but fail to differentiate between Groups 1 and 3 (note the red and green bars). RQUGE, successfully identifies groups violating naturalness (Group 3) and answerability (Group 4), assigns a higher score for Group 2, which lacks complexity, than for Group 1. Our metric, shown in the leftmost bar group, prioritizing essential criteria of a question, can effectively distinguish all four groups of candidates while maintaining the highest rating for the valid questions.
• Figure 2: Case study 1: NACo vs BERTScore. Longest common subsequences between candidate question and RefQ are highlighted.
• Figure 3: Case study 2: NACo vs RQUGE. Context words used by the question are highlighted in the same color if they come from the same passage.
• Figure 4: Correlation with human judgement - Comparing CoT-QA (NACo) with Direct Evaluation
• Figure 5: Examples for each criterion addressed by our metric: Naturalness, Answerability, and Complexity.