Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating the Simulation of Human Personality-Driven Susceptibility to Misinformation with LLMs

Manuel Pratelli, Marinella Petrocchi

TL;DR

This study probes whether LLM agents conditioned on Big-Five personality profiles can mimic human patterns of misinformation susceptibility in news discernment. By mapping each human participant to a personality-conditioned LLM and evaluating the same 24 headlines used in human studies, the authors assess whether trait–discernment associations emerge in synthetic agents. They find that conditioning yields significant shifts in judgments and that GPT-4o closely mirrors human patterns for Agreeableness, Conscientiousness, and Open-Mindedness, though Extraversion and Negative Emotionality show model-specific biases. The work highlights both the promise of scalable, ethically lighter behavioral simulations and the limitations posed by model biases and prompt design, offering a foundation for stress-testing interventions against misinformation while urging caution and further validation. Practically, personality-aligned agents can help prototype resilience strategies and educational tools, but responsible use requires debiasing efforts and cross-cultural verification.

Abstract

Large language models (LLMs) make it possible to generate synthetic behavioural data at scale, offering an ethical and low-cost alternative to human experiments. Whether such data can faithfully capture psychological differences driven by personality traits, however, remains an open question. We evaluate the capacity of LLM agents, conditioned on Big-Five profiles, to reproduce personality-based variation in susceptibility to misinformation, focusing on news discernment, the ability to judge true headlines as true and false headlines as false. Leveraging published datasets in which human participants with known personality profiles rated headline accuracy, we create matching LLM agents and compare their responses to the original human patterns. Certain trait-misinformation associations, notably those involving Agreeableness and Conscientiousness, are reliably replicated, whereas others diverge, revealing systematic biases in how LLMs internalize and express personality. The results underscore both the promise and the limits of personality-aligned LLMs for behavioral simulation, and offer new insight into modeling cognitive diversity in artificial agents.

Evaluating the Simulation of Human Personality-Driven Susceptibility to Misinformation with LLMs

TL;DR

This study probes whether LLM agents conditioned on Big-Five personality profiles can mimic human patterns of misinformation susceptibility in news discernment. By mapping each human participant to a personality-conditioned LLM and evaluating the same 24 headlines used in human studies, the authors assess whether trait–discernment associations emerge in synthetic agents. They find that conditioning yields significant shifts in judgments and that GPT-4o closely mirrors human patterns for Agreeableness, Conscientiousness, and Open-Mindedness, though Extraversion and Negative Emotionality show model-specific biases. The work highlights both the promise of scalable, ethically lighter behavioral simulations and the limitations posed by model biases and prompt design, offering a foundation for stress-testing interventions against misinformation while urging caution and further validation. Practically, personality-aligned agents can help prototype resilience strategies and educational tools, but responsible use requires debiasing efforts and cross-cultural verification.

Abstract

Large language models (LLMs) make it possible to generate synthetic behavioural data at scale, offering an ethical and low-cost alternative to human experiments. Whether such data can faithfully capture psychological differences driven by personality traits, however, remains an open question. We evaluate the capacity of LLM agents, conditioned on Big-Five profiles, to reproduce personality-based variation in susceptibility to misinformation, focusing on news discernment, the ability to judge true headlines as true and false headlines as false. Leveraging published datasets in which human participants with known personality profiles rated headline accuracy, we create matching LLM agents and compare their responses to the original human patterns. Certain trait-misinformation associations, notably those involving Agreeableness and Conscientiousness, are reliably replicated, whereas others diverge, revealing systematic biases in how LLMs internalize and express personality. The results underscore both the promise and the limits of personality-aligned LLMs for behavioral simulation, and offer new insight into modeling cognitive diversity in artificial agents.

Paper Structure

This paper contains 22 sections, 1 equation, 2 figures, 5 tables.

Table of Contents

  1. Present study.
  2. Research questions.
  3. Results.
  4. Contributions.
  5. Related Work
  6. Human Personality Traits Assessment.
  7. Links between Human Traits and Misinformation.
  8. Personality and LLMs.
  9. Methodology
  10. News Discernment and Belief in Misinformation.
  11. Datasets and Materials.
  12. Procedure.
  13. Experimental Settings.
  14. Experiments
  15. Comparing LLM responses with and without personality conditioning.
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Core concept
  • Figure 2: Similarity between human and LLM-based trait-discernment correlations. For each model configuration, we report the cosine similarity between the vector of Pearson correlations (personality traits vs. news discernment) and the corresponding human-derived vector from Calvillo et al. calvillo2021personality (see first row of Table \ref{['tab:links_misinfo_and_traits_with_humans_calvillo']}). LLM-based agent correlation are produced by simulating profiles derived from the same personality dataset calvillo2021personality. The x-axis reports similarity across all traits; the y-axis considers only traits significantly correlated in human data.