Intrinsic Task-based Evaluation for Referring Expression Generation

TL;DR

This paper identifies a limitation of rating-based human evaluations for Referring Expression Generation (REG) and introduces an intrinsic task-based evaluation with two meta-tasks: assessing referential success and prompting rewritings. By applying this protocol to WebNLG REG outputs across five models, the study reveals that performance distinctions emerge more clearly than in prior rating-only studies, with rule-based REG often yielding higher referential success and neural models showing weaker generalization to unseen data. The findings demonstrate that meta-level tasks improve rating reliability and provide richer insight into REG behavior, including how familiarity, rewriting, and context shape referential clarity and coherence. The work advocates for broader use of intrinsic, multi-task evaluation in NLG to better diagnose model strengths and weaknesses and suggests future exploration with large language models and additional corpora.

Abstract

Recently, a human evaluation study of Referring Expression Generation (REG) models had an unexpected conclusion: on \textsc{webnlg}, Referring Expressions (REs) generated by the state-of-the-art neural models were not only indistinguishable from the REs in \textsc{webnlg} but also from the REs generated by a simple rule-based system. Here, we argue that this limitation could stem from the use of a purely ratings-based human evaluation (which is a common practice in Natural Language Generation). To investigate these issues, we propose an intrinsic task-based evaluation for REG models, in which, in addition to rating the quality of REs, participants were asked to accomplish two meta-level tasks. One of these tasks concerns the referential success of each RE; the other task asks participants to suggest a better alternative for each RE. The outcomes suggest that, in comparison to previous evaluations, the new evaluation protocol assesses the performance of each REG model more comprehensively and makes the participants' ratings more reliable and discriminable.

Figures (1)

• Figure 1: An example item in our experiment.