Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility

TL;DR

Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility addresses how to assess LLMs' cognitive plausibility using production-level signals from spontaneous speech. The authors propose two metrics—speech reductions and prosodic prominences—extracted from English, French, and Taiwan Mandarin corpora and test LLMs pre-trained on different genre mixes (conversational, written, mixed) for their ability to predict these signals. Through fine-tuning RoBERTa-based models on token-classification tasks, they show that, after finetuning, models predict these production variables above baselines, with spoken data yielding stronger predictions than written data. The work highlights the value of high-quality speech corpora as benchmarks for evaluating cognitive aspects of LLMs and expands BabyLM's multilingual scope. This provides a complementary, linguistically grounded benchmark that links language-model behavior to human speech production patterns, with implications for developing more cognitively plausible LLMs.

Abstract

The achievements of Large Language Models in Natural Language Processing, especially for high-resource languages, call for a better understanding of their characteristics from a cognitive perspective. Researchers have attempted to evaluate artificial models by testing their ability to predict behavioral (e.g., eye-tracking fixations) and physiological (e.g., brain responses) variables during language processing (e.g., reading/listening). In this paper, we propose using spontaneous speech corpora to derive production variables (speech reductions, prosodic prominences) and applying them in a similar fashion. More precisely, we extract. We then test models trained with a standard procedure on different pretraining datasets (written, spoken, and mixed genres) for their ability to predict these two variables. Our results show that, after some fine-tuning, the models can predict these production variables well above baselines. We also observe that spoken genre training data provides more accurate predictions than written genres. These results contribute to the broader effort of using high-quality speech corpora as benchmarks for LLMs.

Figures (11)

• Figure 1: F-scores on the benchmarks as a function of model (x), task and language (prominence). Model comparisons are based on Bayesian regression analyses where MODEL is the fixed predictor and FOLD is a random intercept. The models were run with weak (uniform) priors using the brms package in R, with post hoc hypothesis testing focused on comparing three small models (Wiki, conversational, mixed). Stars in the figure indicate a one-sided hypothesis with a posterior probability above 95%. Dotted lines correspond to random baselines.
• Figure 2: Perplexity on the benchmarks
• Figure 3: The correlation between perplexity and labels.
• Figure 4: Distribution reduction ratios as calculated in the English Dataset and the threshold selected.
• Figure 5: Distribution reduction ratios as calculated in the French Dataset and the threshold selected.