ErrEval: Error-Aware Evaluation for Question Generation through Explicit Diagnostics

TL;DR

ErrEval introduces an error-aware evaluation framework for question generation by embedding explicit error diagnostics into LLM-based scoring. It defines an 11-type error taxonomy spanning structural, linguistic, and content-related issues, and trains an Error Identifier via iterative refinement to supply diagnostic signals that guide dimension-specific evaluation. Across three benchmarks and multiple evaluators, ErrEval consistently improves alignment with human judgments and reduces overestimation of low-quality questions, demonstrating the value of diagnostic guidance in NLG evaluation. The work enables plug-and-play enhancement of existing evaluators and highlights the practical impact of grounding evaluation in explicit error signals.

Abstract

Automatic Question Generation (QG) often produces outputs with critical defects, such as factual hallucinations and answer mismatches. However, existing evaluation methods, including LLM-based evaluators, mainly adopt a black-box and holistic paradigm without explicit error modeling, leading to the neglect of such defects and overestimation of question quality. To address this issue, we propose ErrEval, a flexible and Error-aware Evaluation framework that enhances QG evaluation through explicit error diagnostics. Specifically, ErrEval reformulates evaluation as a two-stage process of error diagnosis followed by informed scoring. At the first stage, a lightweight plug-and-play Error Identifier detects and categorizes common errors across structural, linguistic, and content-related aspects. These diagnostic signals are then incorporated as explicit evidence to guide LLM evaluators toward more fine-grained and grounded judgments. Extensive experiments on three benchmarks demonstrate the effectiveness of ErrEval, showing that incorporating explicit diagnostics improves alignment with human judgments. Further analyses confirm that ErrEval effectively mitigates the overestimation of low-quality questions.

Figures (14)

• Figure 1: Comparison between vanilla CoT and error-aware evaluation using LLMs, where the former overestimates a flawed question and the latter aligns more closely with human judgment.
• Figure 2: Pilot experiment - Pearson correlation coefficients (%) between model scores and human scores. Flu.: Fluency; Clar.: Clarity; Conc.: Conciseness; Rel.: Relevance; Cons.: Consistency; Ans.: Answerability; AnsC.: Answer Consistency; Avg.: Average.
• Figure 3: Framework of ErrEval. Given a passage ($p$), answer ($a$), generated question ($q$), and evaluation criteria ($c$), ErrEval performs evaluation with explicit error diagnostics. An iteratively trained Error Identifier (EI) detects error types ($el$), which are organized as diagnostic information ($err$) to guide dimension-specific scoring by LLM evaluators. The EI functions as a lightweight and plug-and-play module.
• Figure 4: Effect of EI accuracy on evaluation result across training iterations.
• Figure 5: The overestimation rates of Vanilla Prompt method and ErrEval-large.