Evaluating the Evaluation of Diversity in Natural Language Generation

TL;DR

The paper tackles the lack of principled diversity evaluation in natural language generation by proposing a framework that correlates a proposed diversity metric with a controllable parameter $d$ across contexts. It instantiates two tests—decoding-based (form diversity) and content-based (semantic diversity) evaluation—and demonstrates that humans outperform automatic metrics for content-diversity detection, while decoding mainly affects form rather than meaning. The work introduces a practical methodology for testing and comparing diversity metrics, including the McDiv benchmark and best-practices for crowdsourced human diversity judgments. Collectively, these contributions enable standardized assessment of diversity metrics and guide the development of NLG systems that generate meaningfully diverse outputs.

Abstract

Despite growing interest in natural language generation (NLG) models that produce diverse outputs, there is currently no principled method for evaluating the diversity of an NLG system. In this work, we propose a framework for evaluating diversity metrics. The framework measures the correlation between a proposed diversity metric and a diversity parameter, a single parameter that controls some aspect of diversity in generated text. For example, a diversity parameter might be a binary variable used to instruct crowdsourcing workers to generate text with either low or high content diversity. We demonstrate the utility of our framework by: (a) establishing best practices for eliciting diversity judgments from humans, (b) showing that humans substantially outperform automatic metrics in estimating content diversity, and (c) demonstrating that existing methods for controlling diversity by tuning a "decoding parameter" mostly affect form but not meaning. Our framework can advance the understanding of different diversity metrics, an essential step on the road towards better NLG systems.

Figures (11)

• Figure 1: Diversity metric evaluation: we show two sets of responses to the same question, generated by crowdsourcing workers. While both sets are diverse in terms of form, only set A is diverse in terms of content. Each graph presents the distribution over a diversity metric for sets with high content diversity (blue) and low content diversity (orange). Distributions are approximated over $200$ sets. We observe that the human score metric (absDHS) separates the two distributions, while an n-gram based metric (distinct-n) fails, illustrating that it does not capture content diversity. The dotted lines correspond to the specific sets A and B presented above.
• Figure 2: An overview of our diversity metrics evaluation framework. The tester (machine or human) generates a response set ($\mathcal{S}_{c,d}$) given a diversity parameter ($d$) and a context ($c$). The test score of a metric $m_\text{div}$ is the correlation between the metric score for $\mathcal{S}_{c,d}$ and $d$.
• Figure 3: decTest: Scatter plot of n-gram-based (cosine similarity), neural (BERT-STS) and human (absHDS) metrics as a function of temperature for respGen. Each point corresponds to a single generated set. Error bars of HDS represent the standard deviation over 10 annotator ratings.
• Figure 4: conTest: histograms of metric values of n-gram (distinct n-grams), neural (BERT-Score) and human (absHDS) metrics for promptGen. The orange histogram represents the distribution of the low content diversity class, the blue histogram represents the distribution of the high content diversity class and brown is the intersection between the two. Pointing down triangles represent the threshold $\eta$ of the optimal classifiers. The histograms show how each metric separates the two classes.
• Figure 5: conTest absHDS results depends on the number of ratings per set and the number of sets.