Benchmarking Linguistic Diversity of Large Language Models

TL;DR

This work addresses the gap in evaluating linguistic diversity of large language models beyond traditional quality metrics. It introduces a framework to measure lexical, syntactic, and semantic diversity and benchmarks six LLMs across five NLG tasks, including a focused syntactic case study comparing human and model outputs. Key findings show lexical and semantic diversity align with quality in many tasks, while syntactic diversity often lags and is highly sensitive to training and decoding choices; prompt design has limited impact. The results highlight a risk of homogenization in LLM outputs and call for diversity-aware model design, training, and deployment to preserve human-like linguistic richness in generative systems.

Abstract

The development and evaluation of Large Language Models (LLMs) has primarily focused on their task-solving capabilities, with recent models even surpassing human performance in some areas. However, this focus often neglects whether machine-generated language matches the human level of diversity, in terms of vocabulary choice, syntactic construction, and expression of meaning, raising questions about whether the fundamentals of language generation have been fully addressed. This paper emphasizes the importance of examining the preservation of human linguistic richness by language models, given the concerning surge in online content produced or aided by LLMs. We propose a comprehensive framework for evaluating LLMs from various linguistic diversity perspectives including lexical, syntactic, and semantic dimensions. Using this framework, we benchmark several state-of-the-art LLMs across all diversity dimensions, and conduct an in-depth case study for syntactic diversity. Finally, we analyze how different development and deployment choices impact the linguistic diversity of LLM outputs.

Figures (8)

• Figure 1: Linguistic diversity benchmarking results for NLG tasks detailed in Table \ref{['tab:tasks']}.
• Figure 2: Pearson correlation matrix between diversity metrics and quality metrics.
• Figure 3: Pearson correlation matrix between different diversity metrics.
• Figure 4: Linguistic diversity metrics after different LLM training stages. The pretraining stage is broken into various steps with increasing token counts, which are presented on a log scale for visualization. Experiments are conducted with the OLMo model on the story generation task.
• Figure 5: Impact of instruction tuning.