Large Language Model Recall Uncertainty is Modulated by the Fan Effect

TL;DR

The paper investigates whether large language models exhibit human-like fan effects, linking recall uncertainty to typicality. It employs two paradigms—In-Pretraining Typicality and In-Context Fan Effects—and uses PopulationLM across six base models to measure token-probability-based recall via $P( ext{present})$ and $P( ext{absent})$. Key contributions include the first demonstration of fan effects in LLMs driven by both pretraining-derived typicality and in-context induction, and the finding that mitigating uncertainty disrupts the effect, with model-dependent disruption patterns. The work advances our understanding of cognitive analogies in LLMs and has practical implications for recall-based applications and future human-LLM interaction studies.

Abstract

This paper evaluates whether large language models (LLMs) exhibit cognitive fan effects, similar to those discovered by Anderson in humans, after being pre-trained on human textual data. We conduct two sets of in-context recall experiments designed to elicit fan effects. Consistent with human results, we find that LLM recall uncertainty, measured via token probability, is influenced by the fan effect. Our results show that removing uncertainty disrupts the observed effect. The experiments suggest the fan effect is consistent whether the fan value is induced in-context or in the pre-training data. Finally, these findings provide in-silico evidence that fan effects and typicality are expressions of the same phenomena.

Figures (6)

• Figure 1: Prompt to measure presence/absence belief.
• Figure 2: Top row: Mistral and SOLAR show significant negative Pearson correlations consistent with fan effects across a range of categories. Bottom row: Items present in the context do not elicit a human-like fan effect.
• Figure 3: Left col: Predictions are made using Mistral. Right col: Predictions are made using SOLAR. Bottom row: queried item is present (w/o uncertainty). Top row: queried item is absent (with uncertainty). Fan effects are evident in the negative Pearson correlation (shown in Figure \ref{['fig:IPF']}) in the natural group above the noise floor.
• Figure 4: Prompt to measure presence/absence belief.
• Figure 5: Results of the Anderson recreation experiments on SOLAR and Mistral Top row: queried item is absent with the model predicting true absence (with uncertainty). Bottom row: queried item is present with the model predicting true presence (w/o uncertainty). Lines of best fit are included. Pearson correlations shown in Figure \ref{['fig:ICF-Details']}.