Psychometric Predictive Power of Large Language Models

TL;DR

The paper examines whether instruction tuning and prompting improve models of human reading by comparing psychometric predictive power (PPP) of surprisal from base LLMs, IT-LLMs, and prompt-conditioned variants. It formalizes the linking hypothesis that reading time is related to word surprisal and explores entropy-based extensions, evaluating 26 models on Dundee Corpus and Natural Stories. Results show that base LLMs generally outperform instruction-tuned variants in PPP, Prompts can steer predictions but do not surpass small base models, and metalinguistic prompting yields poor correlation with actual reading times. The findings challenge the notion that instruction tuning or prompting yields cognitive plausibility gains and underscore the continued importance of direct probability outputs for cognitive modeling of reading behavior.

Abstract

Instruction tuning aligns the response of large language models (LLMs) with human preferences. Despite such efforts in human--LLM alignment, we find that instruction tuning does not always make LLMs human-like from a cognitive modeling perspective. More specifically, next-word probabilities estimated by instruction-tuned LLMs are often worse at simulating human reading behavior than those estimated by base LLMs. In addition, we explore prompting methodologies for simulating human reading behavior with LLMs. Our results show that prompts reflecting a particular linguistic hypothesis improve psychometric predictive power, but are still inferior to small base models. These findings highlight that recent advancements in LLMs, i.e., instruction tuning and prompting, do not offer better estimates than direct probability measurements from base LLMs in cognitive modeling. In other words, pure next-word probability remains a strong predictor for human reading behavior, even in the age of LLMs.

Figures (4)

• Figure 1: Comparing the "reading behavior" of humans and LLMs, i.e., reading time from humans is compared with surprisal from LLMs (\ref{['subsec:linking']}). We investigate which surprisal values estimated by: (i) base LLMs, (ii) instruction-tuned (IT) LLMs, (iii) IT-LLMs with prompting, or (iv) IT-LLMs with metalinguistic prompting can better simulate human reading time.
• Figure 2: The relationship between PPL and PPP (see exact scores in Table \ref{['tbl:ppp_ppl']}). Each point corresponds to each LLM, and those with a black edge line are IT-LLMs. The regression line is estimated by base LLMs, and the colored area presents a 95% confidence interval. IT-LLMs were relatively poor (below the line) at balancing PPL and PPP.
• Figure 3: The PPL and PPP of LLMs with prompting are plotted at the top of Figure \ref{['fig:ppl_ppp']}. Each point corresponds to a given combination of LLM and prompt, and those with red-edged outlines are IT-LLMs with a particular prompt. The PPP--PPL regression line is estimated by base LLMs, and the colored area presents the 95% confidence interval. IT-LLMs with prompting are poorer than base LLMs at balancing PPL and PPP.
• Figure 4: The relationship between PPL and PPP when using the prompt: Please complete the following sentence:. Each point corresponds to each LLM, and those with black-edged outlines are IT-LLMs. The PPP--PPL regression line is estimated by base LLMs, and the colored area presents a 95% confidence interval. IT-LLMs were relatively poor (below the line) at balancing PPL and PPP.