Simulating Misinformation Vulnerabilities With Agent Personas

TL;DR

Problem: Real-world experimentation on misinformation is ethically challenging, so the authors propose an agent-based, LLM-driven simulation to study population responses. Approach: they create eight agent personas combining five professions and three mental schemas, and test responses to headlines from the Misinfo Reaction Frames corpus using GPT-4 and LLaMA 3.1 8B Instruct, comparing against ground truth and human judgments. Key findings: mental schemas strongly influence interpretation of misinformation, while professional background plays a lesser role; GPT-based agents align closely with ground-truth labels and human predictions, although sharing propensity diverges. Significance: demonstrates the viability of LLM-driven agents for analyzing trust, polarization, and counter-misinformation strategies at scale in complex information networks.

Abstract

Disinformation campaigns can distort public perception and destabilize institutions. Understanding how different populations respond to information is crucial for designing effective interventions, yet real-world experimentation is impractical and ethically challenging. To address this, we develop an agent-based simulation using Large Language Models (LLMs) to model responses to misinformation. We construct agent personas spanning five professions and three mental schemas, and evaluate their reactions to news headlines. Our findings show that LLM-generated agents align closely with ground-truth labels and human predictions, supporting their use as proxies for studying information responses. We also find that mental schemas, more than professional background, influence how agents interpret misinformation. This work provides a validation of LLMs to be used as agents in an agent-based model of an information network for analyzing trust, polarization, and susceptibility to deceptive content in complex social systems.

Figures (4)

• Figure 1: Overview of simulating reactions to misinformation by different agent personas.
• Figure 2: Heatmap of annotation agreement between LLM-generated agents on identifying whether a news headline constitutes misinformation.
• Figure 3: Comparison of LLM-generated agent predictions to gold labels and human annotator judgments. LLM Model versus gold shows the comparison of each individual LLM-based agent to gold annotations and GPT vs Pred shows the comparison of each individual LLM-based agent to human predictions.
• Figure 4: Bar plot comparing Likert scale ratings of the likelihood to share news headlines, as assessed by LLM agents and human annotators with 1 being not at all likely and five being extremely likely. Agents tend to cluster toward the middle of the Likert scale where humans seem to be more evenly distributed.