Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation

Weiping Fu, Bifan Wei, Jianxiang Hu, Zhongmin Cai, Jun Liu

TL;DR

QGEval tackles the lack of standardized human evaluation in Question Generation by introducing a multi-dimensional benchmark that assesses generated questions across seven dimensions: fluency, clarity, conciseness, relevance, consistency, answerability, and answer consistency. It builds a 3,000-question dataset derived from 200 passages using 15 QG models (including GPT-4 variants) on SQuAD and HotpotQA, following a two-stage generation-and-annotation pipeline. The study reveals that most models struggle with answerability and answer consistency, while many automatic metrics fail to align with human judgments; multi-dimensional, especially reference-free, metrics and even LLM-based evaluators show promise but do not fully replace human evaluation. By providing both the dataset and accompanying evaluation tools, QGEval aims to drive development in QG models and their evaluation, supporting more reliable comparisons and better-aligned automatic metrics. The work highlights a need for richer evaluation criteria and potentially more discriminative dimensions to capture higher-quality, practically useful questions.

Abstract

Automatically generated questions often suffer from problems such as unclear expression or factual inaccuracies, requiring a reliable and comprehensive evaluation of their quality. Human evaluation is widely used in the field of question generation (QG) and serves as the gold standard for automatic metrics. However, there is a lack of unified human evaluation criteria, which hampers consistent and reliable evaluations of both QG models and automatic metrics. To address this, we propose QGEval, a multi-dimensional Evaluation benchmark for Question Generation, which evaluates both generated questions and existing automatic metrics across 7 dimensions: fluency, clarity, conciseness, relevance, consistency, answerability, and answer consistency. We demonstrate the appropriateness of these dimensions by examining their correlations and distinctions. Through consistent evaluations of QG models and automatic metrics with QGEval, we find that 1) most QG models perform unsatisfactorily in terms of answerability and answer consistency, and 2) existing metrics fail to align well with human judgments when evaluating generated questions across the 7 dimensions. We expect this work to foster the development of both QG technologies and their evaluation.

QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation

TL;DR

QGEval tackles the lack of standardized human evaluation in Question Generation by introducing a multi-dimensional benchmark that assesses generated questions across seven dimensions: fluency, clarity, conciseness, relevance, consistency, answerability, and answer consistency. It builds a 3,000-question dataset derived from 200 passages using 15 QG models (including GPT-4 variants) on SQuAD and HotpotQA, following a two-stage generation-and-annotation pipeline. The study reveals that most models struggle with answerability and answer consistency, while many automatic metrics fail to align with human judgments; multi-dimensional, especially reference-free, metrics and even LLM-based evaluators show promise but do not fully replace human evaluation. By providing both the dataset and accompanying evaluation tools, QGEval aims to drive development in QG models and their evaluation, supporting more reliable comparisons and better-aligned automatic metrics. The work highlights a need for richer evaluation criteria and potentially more discriminative dimensions to capture higher-quality, practically useful questions.

Abstract

Automatically generated questions often suffer from problems such as unclear expression or factual inaccuracies, requiring a reliable and comprehensive evaluation of their quality. Human evaluation is widely used in the field of question generation (QG) and serves as the gold standard for automatic metrics. However, there is a lack of unified human evaluation criteria, which hampers consistent and reliable evaluations of both QG models and automatic metrics. To address this, we propose QGEval, a multi-dimensional Evaluation benchmark for Question Generation, which evaluates both generated questions and existing automatic metrics across 7 dimensions: fluency, clarity, conciseness, relevance, consistency, answerability, and answer consistency. We demonstrate the appropriateness of these dimensions by examining their correlations and distinctions. Through consistent evaluations of QG models and automatic metrics with QGEval, we find that 1) most QG models perform unsatisfactorily in terms of answerability and answer consistency, and 2) existing metrics fail to align well with human judgments when evaluating generated questions across the 7 dimensions. We expect this work to foster the development of both QG technologies and their evaluation.
Paper Structure (31 sections, 6 figures, 14 tables)

This paper contains 31 sections, 6 figures, 14 tables.

Table of Contents

  1. Introduction
  2. The QGEval Dataset
  3. Question Generation
  4. Human Evaluation
  5. Evaluation Methodology
  6. Annotation Process
  7. Experiment and Evaluation
  8. Are the seven dimensions appropriate for the evaluation of QG?
  9. How do the QG models perform across the seven dimensions?
  10. Can existing automatic metrics accurately evaluate generated questions?
  11. Automatic Metric
  12. Metric Evaluation
  13. LLM as Evaluator
  14. Related Work
  15. Automatic Metrics
  16. ...and 16 more sections

Figures (6)

  • Figure 1: Pipeline of dataset construction. Stage 1: Generate questions to be evaluated. Stage 2: Conduct two rounds of annotation to form the QGEval dataset.
  • Figure 2: Pearson correlations and p-values of Nemenyi test for seven dimensions.
  • Figure 3: Annotation score distributions across seven dimensions.
  • Figure 4: Annotation interface.
  • Figure 5: Annotation score distributions over seven dimensions.
  • ...and 1 more figures