Predict the Next Word: Humans exhibit uncertainty in this task and language models _____

TL;DR

The paper investigates how well language models calibrate to human uncertainty in next-word prediction by comparing model distributions to human distributions using the Provo Corpus. It formalizes word-level distributions for humans ($p(w|c)$) and models ($q(w|c)$) and evaluates multiple models (GPT-2, BLOOM, ChatGPT) with multi-sample word continuations, showing that standard $ECE$ misrepresents calibration. Using total variation distance ($TVD$) as a direct measure of distributional alignment and additional analyses at syntactic and semantic levels, the study finds substantial gaps between models and human variability, with larger models and RLHF not solving the problem. The results suggest that current training regimes seldom expose models to human production variability, and they advocate for alternative calibration metrics and training data to better capture human uncertainty in language generation.

Abstract

Language models (LMs) are statistical models trained to assign probability to human-generated text. As such, it is reasonable to question whether they approximate linguistic variability exhibited by humans well. This form of statistical assessment is difficult to perform at the passage level, for it requires acceptability judgements (i.e., human evaluation) or a robust automated proxy (which is non-trivial). At the word level, however, given some context, samples from an LM can be assessed via exact matching against a prerecorded dataset of alternative single-word continuations of the available context. We exploit this fact and evaluate the LM's ability to reproduce variability that humans (in particular, a population of English speakers) exhibit in the 'next word prediction' task. This can be seen as assessing a form of calibration, which, in the context of text classification, Baan et al. (2022) termed calibration to human uncertainty. We assess GPT2, BLOOM and ChatGPT and find that they exhibit fairly low calibration to human uncertainty. We also verify the failure of expected calibration error (ECE) to reflect this, and as such, advise the community against relying on it in this setting.

Figures (34)

• Figure 1: Estimated human and model distributions for contexts (15 most probable words of each distribution).
• Figure 2: KDE plot of TVD values between a model and the estimated human target cpd, and between oracles.
• Figure 3: Regression coefficients and their credible intervals. Features, in order: Human entropy, Model Entropy, TVD between oracles, Context Length, Punctuation, and 10 universal POS tags.
• Figure 4: Histogram of TVD$_{syn}$ and TVD$_{sem}$ for all contexts
• Figure 5: Histogram of TVD values for (biased) model and oracle distributions when compared to the full human distribution