Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can LLMs Simulate Human Behavioral Variability? A Case Study in the Phonemic Fluency Task

Mengyang Qiu, Zoe Brisebois, Siena Sun

TL;DR

The paper interrogates whether large language models can simulate human behavioral variability in a phonemic fluency task. By evaluating 34 configurations across six providers and comparing outputs to 106 human participants, the study shows that while some LLMs can match average performance, they fail to reproduce human variability and the retrieval structures evident in human memory search, with Claude 3.7 Sonnet being the closest to human averages. Item-level analyses reveal similar but attenuated associations with lexical variables and Zipfian distributions, and network analyses identify fundamental differences in how humans and LLMs organize word retrieval. The results highlight critical limitations in using LLMs as proxies for human cognition in behavioral research and suggest that current approaches, including prompts, temperature adjustments, and ensembles, are insufficient to capture human-like variability; future work should pursue persona-based or cognitive-profile modeling to better emulate individual differences.

Abstract

Large language models (LLMs) are increasingly explored as substitutes for human participants in cognitive tasks, but their ability to simulate human behavioral variability remains unclear. This study examines whether LLMs can approximate individual differences in the phonemic fluency task, where participants generate words beginning with a target letter. We evaluated 34 model configurations, varying prompt specificity, sampling temperature, and model type, and compared outputs to responses from 106 human participants. While some configurations, especially Claude 3.7 Sonnet, matched human averages and lexical preferences, none reproduced the scope of human variability. LLM outputs were consistently less diverse and structurally rigid, and LLM ensembles failed to increase diversity. Network analyses further revealed fundamental differences in retrieval structure between humans and models. These results highlight key limitations in using LLMs to simulate human cognition and behavior.

Can LLMs Simulate Human Behavioral Variability? A Case Study in the Phonemic Fluency Task

TL;DR

The paper interrogates whether large language models can simulate human behavioral variability in a phonemic fluency task. By evaluating 34 configurations across six providers and comparing outputs to 106 human participants, the study shows that while some LLMs can match average performance, they fail to reproduce human variability and the retrieval structures evident in human memory search, with Claude 3.7 Sonnet being the closest to human averages. Item-level analyses reveal similar but attenuated associations with lexical variables and Zipfian distributions, and network analyses identify fundamental differences in how humans and LLMs organize word retrieval. The results highlight critical limitations in using LLMs as proxies for human cognition in behavioral research and suggest that current approaches, including prompts, temperature adjustments, and ensembles, are insufficient to capture human-like variability; future work should pursue persona-based or cognitive-profile modeling to better emulate individual differences.

Abstract

Large language models (LLMs) are increasingly explored as substitutes for human participants in cognitive tasks, but their ability to simulate human behavioral variability remains unclear. This study examines whether LLMs can approximate individual differences in the phonemic fluency task, where participants generate words beginning with a target letter. We evaluated 34 model configurations, varying prompt specificity, sampling temperature, and model type, and compared outputs to responses from 106 human participants. While some configurations, especially Claude 3.7 Sonnet, matched human averages and lexical preferences, none reproduced the scope of human variability. LLM outputs were consistently less diverse and structurally rigid, and LLM ensembles failed to increase diversity. Network analyses further revealed fundamental differences in retrieval structure between humans and models. These results highlight key limitations in using LLMs to simulate human cognition and behavior.

Paper Structure

This paper contains 16 sections, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Human data source
  4. LLM simulation procedure
  5. Participant-Level Analysis
  6. Number of responses
  7. Variability of responses
  8. Item-Level Analysis
  9. Distribution of production frequency
  10. Linguistic variables and production frequency
  11. Network Analysis
  12. Network construction approach
  13. Structural comparison of human and Claude networks
  14. Discussion
  15. Overview of LLM Providers, Prompts, and Sampling Temperatures
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Full prompt used to simulate the F fluency task, including participant age, education, and number of correct responses, with instructions identical to those given to human participants
  • Figure 2: Clustering coefficient and average shortest path length for human and Claude networks based on 1,000 bootstrap subnetworks (50% nodes)