Generalized Measures of Anticipation and Responsivity in Online Language Processing

TL;DR

This framework provides a formal definition of anticipatory and responsive measures, and it equips experimenters with the tools to define new, more expressive measures beyond standard next-symbol entropy and surprisal.

Abstract

We introduce a generalization of classic information-theoretic measures of predictive uncertainty in online language processing, based on the simulation of expected continuations of incremental linguistic contexts. Our framework provides a formal definition of anticipatory and responsive measures, and it equips experimenters with the tools to define new, more expressive measures beyond standard next-symbol entropy and surprisal. While extracting these standard quantities from language models is convenient, we demonstrate that using Monte Carlo simulation to estimate alternative responsive and anticipatory measures pays off empirically: New special cases of our generalized formula exhibit enhanced predictive power compared to surprisal for human cloze completion probability as well as ELAN, LAN, and N400 amplitudes, and greater complementarity with surprisal in predicting reading times.

Figures (18)

• Figure 1: Coefficient of variation (top), correlation between resamples (center), and runtimes (bottom) for sampling-based measures across the stimuli in the Aligned dataset. Confidence intervals ($95\%$) are too narrow to be visible; the horizontal axis is in log scale. The average runtime for the exact metrics (surprisal, probability, expected next-symbol surprisal, and expected next-symbol probability) is $0.002$ seconds.
• Figure 2: Predictive power (${ \Delta_{{ R^2}}}$) of responsive generalized surprisal models for event-related potentials and reading times. 95% confidence intervals. Significance color-coded: blue for $p<0.0001$, gray for $p>0.01$.
• Figure 3: Probability and surprisal against human cloze probabilities and predictability ratings, with Pearson correlation coefficients $r$ and regression lines. For regressions, $x^{({ m})} = { \gamma}({ \boldsymbol{w}}^{({ m})}, {\boldsymbol{c}}^{({ m})})$ is the predictor and $y^{({ m})} = { \psi}({ \boldsymbol{w}}^{({ m})}, {\boldsymbol{c}}^{({ m})})$ the predicted variable.
• Figure 4: Difference between the predictive power of anticipatory measures used in combination with surprisal vs. expected next-symbol surprisal used in combination with surprisal; 95% confidence intervals. The red dotted line represents the combined baseline regressor. Significance is color-coded, as described in the legend.
• Figure 5: Pearson correlation between responsive and anticipatory measures. Estimates obtained for the Aligned dataset. Monte Carlo (MC) samples with ${ N}=2^{9}$ and ${ L} = 5$ from GPT-2 Small.

Theorems & Definitions (2)

• Definition 1: Generalized Surprisal
• Definition 2: Anticipation and Responsivity