Order in the Evaluation Court: A Critical Analysis of NLG Evaluation Trends

TL;DR

Order in the Evaluation Court analyzes NLG evaluation practices across 14,171 papers from ACL, EMNLP, NAACL, and INLG (2020–2025) using a multi‑LLM information extraction pipeline. It reveals pervasive metric inertia toward legacy, task‑agnostic metrics, and a rapid rise of LLMs as judges (LaaJ) that are not consistently validated against human judgments. The study shows that LaaJ and human evaluations emphasize different signals and finds only limited direct comparisons between the two, highlighting a need for explicit validation and a stratified evaluation protocol. Practically, the work provides concrete recommendations to modernize leaderboards, map metrics to criteria, and separate judge‑like signals from human judgments to improve NLG evaluation rigor.

Abstract

Despite advances in Natural Language Generation (NLG), evaluation remains challenging. Although various new metrics and LLM-as-a-judge (LaaJ) methods are proposed, human judgment persists as the gold standard. To systematically review how NLG evaluation has evolved, we employ an automatic information extraction scheme to gather key information from NLG papers, focusing on different evaluation methods (metrics, LaaJ and human evaluation). With extracted metadata from 14,171 papers across four major conferences (ACL, EMNLP, NAACL, and INLG) over the past six years, we reveal several critical findings: (1) Task Divergence: While Dialogue Generation demonstrates a rapid shift toward LaaJ (>40% in 2025), Machine Translation remains locked into n-gram metrics, and Question Answering exhibits a substantial decline in the proportion of studies conducting human evaluation. (2) Metric Inertia: Despite the development of semantic metrics, general-purpose metrics (e.g., BLEU, ROUGE) continue to be widely used across tasks without empirical justification, often lacking the discriminative power to distinguish between specific quality criteria. (3) Human-LaaJ Divergence: Our association analysis challenges the assumption that LLMs act as mere proxies for humans; LaaJ and human evaluations prioritize very different signals, and explicit validation is scarce (<8% of papers comparing the two), with only moderate to low correlation. Based on these observations, we derive practical recommendations to improve the rigor of future NLG evaluation.

Figures (9)

• Figure 1: The fractured landscape of NLG evaluation, size of dots indicates usage frequency. Metrics are mapped by Likelihood Ratio (LR) to Human Eval ($x$) vs. LaaJ ($y$), in log-scale. The bottom left gray bubbles reveal "metric inertia" around generic metrics like BLEU. In contrast, highly specific metrics ($LR > 5$) diverge: orange bubbles highlight metrics associated with highly LaaJ and blue bubbles metrics with highly human associations, indicating that LaaJ is not a direct proxy for human judgment.
• Figure 2: Our paper annotation pipeline (§ \ref{['sec:annotation']}), including converting PDF to text (§ \ref{['sec:extraction']}), extraction of metadata based on the NLG evaluation questionnaire (Table \ref{['tab:llm-schema']}), harmonization based on majority vote, filtering, and merging (§ \ref{['sec:merging']}), and human validation (§ \ref{['sec:human_annotation']}), yielding 14k structured papers including a subset of 110 papers with gold annotations.
• Figure 3: Distribution of different terms across years, numbers are counted are after normalization. The only abnormal trend is the number of languages, which decreased from 2024-2025.
• Figure 4: Distribution of papers (2020 -- 2025) with different evaluation methods across the top-four tasks.
• Figure 5: Bump charts of task-specific NLG evaluation trends across four tasks (left-to-right) and three paradigms (automatic, human, LaaJ) (top-down). The metrics and criteria are ranked by their task association, and size indicates the frequency of that metric among papers of that task. Only top-10 most frequent metrics and criteria are included.