Is Reference Necessary in the Evaluation of NLG Systems? When and Where?

TL;DR

This work critically compares reference-based and reference-free automatic metrics for evaluating NLG outputs across eight datasets and eight evaluation models. Using meta-evaluation methods—correlations with human judgments, criterion-level analysis with perturbations and KS tests, and stability analyses—the study shows reference-free metrics often correlate more strongly with humans and better detect language deficiencies, though their effectiveness varies by task and candidate quality. The findings support using reference-free metrics for many tasks (notably summarization and data-to-text) while highlighting the need for task-specific validation prior to deployment, especially for inputs with unusual structure or highly diverse answer spaces. The paper offers practical guidance for selecting and pre-validating metrics and discusses limitations, including stability concerns and language-task generalizability, pointing to future work in developing robust, task-agnostic evaluation tools.

Abstract

The majority of automatic metrics for evaluating NLG systems are reference-based. However, the challenge of collecting human annotation results in a lack of reliable references in numerous application scenarios. Despite recent advancements in reference-free metrics, it has not been well understood when and where they can be used as an alternative to reference-based metrics. In this study, by employing diverse analytical approaches, we comprehensively assess the performance of both metrics across a wide range of NLG tasks, encompassing eight datasets and eight evaluation models. Based on solid experiments, the results show that reference-free metrics exhibit a higher correlation with human judgment and greater sensitivity to deficiencies in language quality. However, their effectiveness varies across tasks and is influenced by the quality of candidate texts. Therefore, it's important to assess the performance of reference-free metrics before applying them to a new task, especially when inputs are in uncommon form or when the answer space is highly variable. Our study can provide insight into the appropriate application of automatic metrics and the impact of metric choice on evaluation performance.

Figures (10)

• Figure 1: Evaluation mechanism of automatic evaluation metrics. Reference-based metrics measure the similarity between $hyp$ and $ref$s, while reference-free metrics instead measure how likely the $hyp$ is in the derived space $\mathcal{D}$.
• Figure 2: Coherence
• Figure 3: Consistency
• Figure 4: Fluency
• Figure 6: Coherence