Persuasion at Play: Understanding Misinformation Dynamics in Demographic-Aware Human-LLM Interactions

TL;DR

The paper investigates bidirectional misinformation dynamics in demographic-aware human-LLM interactions using the PANDORA framework, which analyzes LLM-to-human and human-to-LLM persuasion, as well as multi-agent LLM interactions across demographic personas. It combines three misinfo datasets (FN, RE, SS) with three LLMs (gpt-35-turbo, llama-3-70b-instruct, qwen-2.5-72B-instruct) to quantify correctness rates and persuasion effects, employing demographic groupings (rural/urban, female/male, young/old). Key findings show that demographics shape susceptibility in both humans and LLM personas, multi-agent LLMs exhibit echo-chamber dynamics in homogeneous groups, and LLM-generated persuasion can boost correctness while human persuasion may dampen it in multi-agent settings. These insights inform targeted interventions and responsible deployment of LLMs for misinformation mitigation, and the authors publicly release the PANDORA framework for reproducible research.

Abstract

Existing challenges in misinformation exposure and susceptibility vary across demographic groups, as some populations are more vulnerable to misinformation than others. Large language models (LLMs) introduce new dimensions to these challenges through their ability to generate persuasive content at scale and reinforcing existing biases. This study investigates the bidirectional persuasion dynamics between LLMs and humans when exposed to misinformative content. We analyze human-to-LLM influence using human-stance datasets and assess LLM-to-human influence by generating LLM-based persuasive arguments. Additionally, we use a multi-agent LLM framework to analyze the spread of misinformation under persuasion among demographic-oriented LLM agents. Our findings show that demographic factors influence susceptibility to misinformation in LLMs, closely reflecting the demographic-based patterns seen in human susceptibility. We also find that, similar to human demographic groups, multi-agent LLMs exhibit echo chamber behavior. This research explores the interplay between humans and LLMs, highlighting demographic differences in the context of misinformation and offering insights for future interventions.

Figures (17)

• Figure 1: In our study, we investigate the differences in persuasion effects of LLMs on humans, and of humans on LLMs. To assess the impact of persuasion, we conduct experiments involving human participants from diverse demographic groups---varying by age, gender, and geographical backgrounds; and LLMs with different demographic persona.
• Figure 2: Multi-Agent LLM Architecture: Homogeneous and Heterogeneous groups engage in interaction rounds to decide if a news item is true or false. They are provided with persuasion texts during the interaction. Note that n=4 for our experiments.
• Figure 3: Human annotation guidelines. Stances are generated by LLMs.
• Figure 4: LLM-to-Human Persuasion: Correctness rates across different human demographics RE and FN
• Figure 5: Human-to-LLM Persuasion: Correctness rates for different model demographics for RE and SS.