Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the scaling relationship between cloze probabilities and language model next-token prediction

Cassandra L. Jacobs, Morgan Grobol

TL;DR

The results provide support for the claim that the greater memorization capacity of larger models helps them guess more semantically appropriate words, but makes them less sensitive to low-level information that is relevant for word recognition.

Abstract

Recent work has shown that larger language models have better predictive power for eye movement and reading time data. While even the best models under-allocate probability mass to human responses, larger models assign higher-quality estimates of next tokens and their likelihood of production in cloze data because they are less sensitive to lexical co-occurrence statistics while being better aligned semantically to human cloze responses. The results provide support for the claim that the greater memorization capacity of larger models helps them guess more semantically appropriate words, but makes them less sensitive to low-level information that is relevant for word recognition.

On the scaling relationship between cloze probabilities and language model next-token prediction

TL;DR

The results provide support for the claim that the greater memorization capacity of larger models helps them guess more semantically appropriate words, but makes them less sensitive to low-level information that is relevant for word recognition.

Abstract

Recent work has shown that larger language models have better predictive power for eye movement and reading time data. While even the best models under-allocate probability mass to human responses, larger models assign higher-quality estimates of next tokens and their likelihood of production in cloze data because they are less sensitive to lexical co-occurrence statistics while being better aligned semantically to human cloze responses. The results provide support for the claim that the greater memorization capacity of larger models helps them guess more semantically appropriate words, but makes them less sensitive to low-level information that is relevant for word recognition.
Paper Structure (16 sections, 5 equations, 6 figures, 3 tables)

This paper contains 16 sections, 5 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Cloze data
  3. Experiment 1: Comparing production probabilities
  4. Cloze probabilities
  5. Cloze log-odds
  6. Rank correlations
  7. Experiment 2: Model capacity and human cloze alignment
  8. Pythia correlation to n-gram baseline
  9. Pythia correlation to cloze completions
  10. Experiment 3: Human-LM semantic representational similarity analysis
  11. Analysis procedure
  12. Positive pointwise mutual information to assess shared response biases
  13. Semantic neighborhood structure of contextual embedding representations
  14. Discussion
  15. Limitations
  16. ...and 1 more sections

Figures (6)

  • Figure 1: Human-NLM cloze correlations for Pythia-2.8B-deduped showing major deficiencies in assigning probability mass to human completions. All models at all sizes showed similar patterns and are not pictured here.
  • Figure 2: Relationship between language model ranks and cloze ranks. Error bars represent bootstrapped confidence intervals.
  • Figure 3: Correlation strength between Wikipedia 5-gram scores and Pythia next-token probabilities by model size and data deduplication. X axis in log scale.
  • Figure 4: Effect of model capacity and size on cloze correlation.
  • Figure 5: Correlation between model capacity, PCA dimensionality, and correlation between language model and human semantic spaces.
  • ...and 1 more figures