Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices

Patrícia Schmidtová, Saad Mahamood, Simone Balloccu, Ondřej Dušek, Albert Gatt, Dimitra Gkatzia, David M. Howcroft, Ondřej Plátek, Adarsa Sivaprasad

TL;DR

This paper surveys current automatic evaluation practices in natural language generation by annotating 110 2023 papers from INLG and ACL Generation to understand metric usage, rationale, and reporting. It reveals heavy reliance on reference-based metrics like BLEU and ROUGE, widespread absence of justification for metric choices, and limited reporting of implementation details, which hinders reproducibility. The study also documents limited alignment between automatic metrics and human judgments, uneven code and data release practices, and a proliferation of new trainable metrics focused on factuality. Based on these findings, the authors propose concrete recommendations to improve evaluation quality, reproducibility, and transparency, including explicit rationales for metric choices, careful metric combinations, and mandatory sharing of code and data. The work contributes a dataset of annotated papers and actionable guidelines intended to raise standards for rigorous NLG evaluation going forward.

Abstract

Automatic metrics are extensively used to evaluate natural language processing systems. However, there has been increasing focus on how they are used and reported by practitioners within the field. In this paper, we have conducted a survey on the use of automatic metrics, focusing particularly on natural language generation (NLG) tasks. We inspect which metrics are used as well as why they are chosen and how their use is reported. Our findings from this survey reveal significant shortcomings, including inappropriate metric usage, lack of implementation details and missing correlations with human judgements. We conclude with recommendations that we believe authors should follow to enable more rigour within the field.

Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices

TL;DR

This paper surveys current automatic evaluation practices in natural language generation by annotating 110 2023 papers from INLG and ACL Generation to understand metric usage, rationale, and reporting. It reveals heavy reliance on reference-based metrics like BLEU and ROUGE, widespread absence of justification for metric choices, and limited reporting of implementation details, which hinders reproducibility. The study also documents limited alignment between automatic metrics and human judgments, uneven code and data release practices, and a proliferation of new trainable metrics focused on factuality. Based on these findings, the authors propose concrete recommendations to improve evaluation quality, reproducibility, and transparency, including explicit rationales for metric choices, careful metric combinations, and mandatory sharing of code and data. The work contributes a dataset of annotated papers and actionable guidelines intended to raise standards for rigorous NLG evaluation going forward.

Abstract

Automatic metrics are extensively used to evaluate natural language processing systems. However, there has been increasing focus on how they are used and reported by practitioners within the field. In this paper, we have conducted a survey on the use of automatic metrics, focusing particularly on natural language generation (NLG) tasks. We inspect which metrics are used as well as why they are chosen and how their use is reported. Our findings from this survey reveal significant shortcomings, including inappropriate metric usage, lack of implementation details and missing correlations with human judgements. We conclude with recommendations that we believe authors should follow to enable more rigour within the field.
Paper Structure (84 sections, 16 figures, 4 tables)

This paper contains 84 sections, 16 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Evaluation Surveys in NLG
  3. Survey Method
  4. Paper selection
  5. Annotation procedure
  6. Iterative refinement and inter-annotator agreement
  7. Analysis and Results
  8. Metric-Level Analysis
  9. Frame of comparison:
  10. BLEU and ROUGE:
  11. Trainable metrics
  12. Metric Rationales:
  13. New Metrics:
  14. Appendix:
  15. Automatic vs. Human Evaluations
  16. ...and 69 more sections

Figures (16)

  • Figure 1: Usage percentages of top 10 metric families in INLG and ACL, with metric category color-coded.
  • Figure 2: The percentage of automatic metric types used in both INLG and ACL conferences.
  • Figure 3: Co-occurrence of types of rationales with the authors correlating the metric results to human judgment.
  • Figure 4: The percentage of papers that state a form of correlation between their automatic and human evaluation results.
  • Figure 5: Distributions of different metric families used to evaluate a given task across ACL and INLG (with percentages of metric usages for the given task on top and absolute counts below).
  • ...and 11 more figures