Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Replicating Human Motivated Reasoning Studies with LLMs

Neeley Pate, Adiba Mahbub Proma, Hangfeng He, James N. Druckman, Daniel Molden, Gourab Ghoshal, Ehsan Hoque

TL;DR

This work investigates whether base, persona-free LLMs replicate human motivated reasoning by replicating four political studies across seven models. It finds that base LLMs do not emulate human directional or accuracy-driven reasoning, showing weak or absent correlations with human responses and limited ability to assess argument strength. A key pattern is that LLMs exhibit reduced response variability compared to humans, suggesting they do not capture the diversity of human opinions under motivational manipulations. The results caution researchers using LLMs as survey proxies or for automatic argument assessment, and point to the need for personas, advanced prompting, or hybrid human–machine approaches to better align AI with human reasoning in these tasks.

Abstract

Motivated reasoning -- the idea that individuals processing information may be motivated to reach a certain conclusion, whether it be accurate or predetermined -- has been well-explored as a human phenomenon. However, it is unclear whether base LLMs mimic these motivational changes. Replicating 4 prior political motivated reasoning studies, we find that base LLM behavior does not align with expected human behavior. Furthermore, base LLM behavior across models shares some similarities, such as smaller standard deviations and inaccurate argument strength assessments. We emphasize the importance of these findings for researchers using LLMs to automate tasks such as survey data collection and argument assessment.

Replicating Human Motivated Reasoning Studies with LLMs

TL;DR

This work investigates whether base, persona-free LLMs replicate human motivated reasoning by replicating four political studies across seven models. It finds that base LLMs do not emulate human directional or accuracy-driven reasoning, showing weak or absent correlations with human responses and limited ability to assess argument strength. A key pattern is that LLMs exhibit reduced response variability compared to humans, suggesting they do not capture the diversity of human opinions under motivational manipulations. The results caution researchers using LLMs as survey proxies or for automatic argument assessment, and point to the need for personas, advanced prompting, or hybrid human–machine approaches to better align AI with human reasoning in these tasks.

Abstract

Motivated reasoning -- the idea that individuals processing information may be motivated to reach a certain conclusion, whether it be accurate or predetermined -- has been well-explored as a human phenomenon. However, it is unclear whether base LLMs mimic these motivational changes. Replicating 4 prior political motivated reasoning studies, we find that base LLM behavior does not align with expected human behavior. Furthermore, base LLM behavior across models shares some similarities, such as smaller standard deviations and inaccurate argument strength assessments. We emphasize the importance of these findings for researchers using LLMs to automate tasks such as survey data collection and argument assessment.
Paper Structure (38 sections, 9 figures, 19 tables)

This paper contains 38 sections, 9 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Motivated Reasoning
  4. Bias in Persona-Based and Non-Persona-Based LLMs
  5. Human Study Selection
  6. Study 1: The Influence of Partisan Motivated Reasoning on Public Opinion
  7. Study 2: Counteracting the Politicization of Science
  8. Study 3: How Elite Partisan Polarization Affects Public Opinion Formation
  9. Study 4: Partisanship and Preference Formation: Competing Motivations, Elite Polarization, and Issue Importance
  10. Experiments and Evaluation
  11. Experimental Setup
  12. LLM Selection
  13. Human Study Replication
  14. Evaluation Setup
  15. Calculating LLM Averages
  16. ...and 23 more sections

Figures (9)

  • Figure 1: Graphic Describing the Procedure Followed in the Study. An example of different types of prompts (accuracy vs. directional) is shown to highlight potential differences in reasoning.
  • Figure 2: Distribution of Standard Deviations Between the LLM and Human Responses on Opinion Formation.
  • Figure 3: Distribution of Standard Deviations Between the LLM and Human Responses on Pro - Con Assessment.
  • Figure 4: Results for the average percent change in support compared to the control, partitioned by model and condition, compared against the reported human percent change in support (Study 1). Human results from the original paper provided in orange for reader convenience.
  • Figure 5: Results for the average percent of support, partitioned by model and condition, compared against the reported human percent change in support (Study 2). Human results from the original paper provided in orange for reader convenience.
  • ...and 4 more figures