Predicting Biased Human Decision-Making with Large Language Models in Conversational Settings

TL;DR

This work examines whether large language models (LLMs) can predict biased human decision-making in conversational settings and reproduce load–bias interactions observed in humans. It combines two preregistered human experiments (six decision problems under simple vs complex prior dialogue) with a cross-model LLM evaluation (demographics plus prior dialogue) to predict individual choices and population bias patterns. Across models, GPT-4.1 most closely aligned with human biases and, in some cases, captured load-related effects when prompted to simulate biases (HL3), though this sometimes inflated effects beyond human levels. The findings suggest LLMs can serve as scalable simulators for bias-aware human–computer interaction research and design, while underscoring limitations around mechanism vs pattern matching and the need for ethical safeguards in adaptive conversational systems.

Abstract

We examine whether large language models (LLMs) can predict biased decision-making in conversational settings, and whether their predictions capture not only human cognitive biases but also how those effects change under cognitive load. In a pre-registered study (N = 1,648), participants completed six classic decision-making tasks via a chatbot with dialogues of varying complexity. Participants exhibited two well-documented cognitive biases: the Framing Effect and the Status Quo Bias. Increased dialogue complexity resulted in participants reporting higher mental demand. This increase in cognitive load selectively, but significantly, increased the effect of the biases, demonstrating the load-bias interaction. We then evaluated whether LLMs (GPT-4, GPT-5, and open-source models) could predict individual decisions given demographic information and prior dialogue. While results were mixed across choice problems, LLM predictions that incorporated dialogue context were significantly more accurate in several key scenarios. Importantly, their predictions reproduced the same bias patterns and load-bias interactions observed in humans. Across all models tested, the GPT-4 family consistently aligned with human behavior, outperforming GPT-5 and open-source models in both predictive accuracy and fidelity to human-like bias patterns. These findings advance our understanding of LLMs as tools for simulating human decision-making and inform the design of conversational agents that adapt to user biases.

Figures (8)

• Figure 1: Experimental procedure outlining the sequence of tasks. Participants were first introduced to the dialogue (Simple or Complex), followed by the assigned condition (Framing: Framed vs. Alternative Framing; Status Quo: Neutral, A, or B). After completing the choice problem, participants filled out the NASA-TLX and post-task questionnaires (recall, familiarity, attention checks, and tool usage).
• Figure 2: Deviation in accuracy from GPT-4.1 reference model across Human-Likeness (HL1, HL2, HL3) levels for each LLM. Positive values indicate higher accuracy than GPT-4.1; negative values indicate lower accuracy. gpt4_1_blrp stands for GPT-4.1 baseline experiment with human response perturbation.
• Figure 3: Boxplots, Effect Sizes, Significances ($*** p < 0.001$, ns - no significance), and Means of NASA-TLX Scores for Simple vs. Complex Task Conditions.
• Figure 4: NASA-TLX scores show significantly higher perceived mental demand and effort under the Complex Dialogue condition, confirming the effectiveness of the cognitive load manipulation.
• Figure 5: Scatterplots with regression lines showing associations between Mental Demand and Memory Task Accuracy (left), Response Time and Memory Task Accuracy (center), and Response Time and Mental Demand (right), the first two under Load condition. Shaded bands represent 95% confidence intervals.