Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal

TL;DR

This paper addresses how predictability is quantified in language processing by comparing cloze-based probabilities with LM-based surprisal (notably from GPT-2) as predictors of word-by-word reading times. It demonstrates that LM surprisal generally provides a superior fit to processing difficulty, and then tests three mechanisms—resolution, semantic discrimination, and low-frequency handling—that could underlie this advantage by manipulating GPT-2 probabilities and observing effects on RT predictions. The results support all three hypotheses, showing that LM surprisal benefits from high-resolution probability estimates, fine-grained semantic distinctions, and inclusion of low-frequency continuations, while constraining any of these factors diminishes predictive power. The findings advocate methodological triangulation to improve cloze studies and prompt further work on whether human prediction is as sensitive as LM probabilities to these fine-grained distinctions, with implications for cross-linguistic generalizability and theoretical accounts of prediction in language comprehension. The core metric, LM surprisal, is defined as $-\,\log_2 P(w_t|context)$, and the work emphasizes controlled comparisons and cross-validation to quantify predictive power across datasets.

Abstract

How predictable a word is can be quantified in two ways: using human responses to the cloze task or using probabilities from language models (LMs).When used as predictors of processing effort, LM probabilities outperform probabilities derived from cloze data. However, it is important to establish that LM probabilities do so for the right reasons, since different predictors can lead to different scientific conclusions about the role of prediction in language comprehension. We present evidence for three hypotheses about the advantage of LM probabilities: not suffering from low resolution, distinguishing semantically similar words, and accurately assigning probabilities to low-frequency words. These results call for efforts to improve the resolution of cloze studies, coupled with experiments on whether human-like prediction is also as sensitive to the fine-grained distinctions made by LM probabilities.

Figures (7)

• Figure 1: How predictable a word is in its context has traditionally been quantified using the cloze task (red), which is increasingly being replaced with LM probabilities in recent years (blue). After establishing that LM probabilities generally provide a better predictor of reading times than cloze responses (Experiment 1), we test three hypotheses about why (yellow, green, magenta) by conducting a hypothesis-driven manipulation of LM probabilities (Experiment 2).
• Figure 2: Increase in per-observation log likelihood over the baseline regression models due to including cloze surprisal, GPT2 surprisal, and both predictors, averaged over the 10 folds used in cross-validation. Error bars denote one standard error of the mean (SEM) across the 10 folds. Among the two comparisons of interest (Cloze vs. Both; GPT2 vs. Both), differences that achieve significance at the $0.05$ level by a paired permutation test under a 12-way Bonferroni correction (two comparisons on six measures) are marked with an asterisk.
• Figure 3: (Top) Increase in per-observation log likelihood over the baseline regression models due to including cloze surprisal, GPT2 surprisal, and manipulated variants of GPT2 surprisal, averaged over all 60 folds (10 folds of six measures) used in cross-validation. Error bars denote one SEM across all 60 folds. Among the three comparisons between GPT2 and its manipulated variants, differences that achieve significance at the $0.05$ level by a paired permutation test under a 3-way Bonferroni correction are marked with an asterisk. (Bottom) Pearson correlation between cloze probabilities and each set of GPT2-based probabilities, calculated over the three text corpora. Error bars denote 95% confidence intervals derived by a permutation test.
• Figure 4: Increase in per-observation log likelihood in $10^{-4}$ nats over the baseline regression models due to including cloze surprisal, manipulated GPT2 surprisal, and both predictors, averaged over the 10 folds used in cross-validation. Error bars denote one SEM across the 10 folds. Among the two comparisons of interest (Cloze vs. Both; GPT2-H$_{\{1,2,3\}}$ vs. Both), differences that achieve significance at the $0.05$ level by a paired permutation test under a 12-way Bonferroni correction (two comparisons on six measures) are marked with an asterisk.
• Figure 5: Increase in per-observation log likelihood, averaged over all folds used in cross-validation. Error bars denote one SEM across all folds. Differences that achieve significance at the $0.05$ level under a 10-way Bonferroni correction (two comparisons on five measures) are marked with an asterisk.