Simulating Online Social Media Conversations on Controversial Topics Using AI Agents Calibrated on Real-World Data

Abstract

Online social networks offer a valuable lens to analyze both individual and collective phenomena. Researchers often use simulators to explore controlled scenarios, and the integration of Large Language Models (LLMs) makes these simulations more realistic by enabling agents to understand and generate natural language content. In this work, we investigate the behavior of LLM-based agents in a simulated microblogging social network. We initialize agents with realistic profiles calibrated on real-world online conversations from the 2022 Italian political election and extend an existing simulator by introducing mechanisms for opinion modeling. We examine how LLM agents simulate online conversations, interact with others, and evolve their opinions under different scenarios. Our results show that LLM agents generate coherent content, form connections, and build a realistic social network structure. However, their generated content displays less heterogeneity in tone and toxicity compared to real data. We also find that LLM-based opinion dynamics evolve over time in ways similar to traditional mathematical models. Varying parameter configurations produces no significant changes, indicating that simulations require more careful cognitive modeling at initialization to replicate human behavior more faithfully. Overall, we demonstrate the potential of LLMs for simulating user behavior in social environments, while also identifying key challenges in capturing heterogeneity and complex dynamics.

Figures (14)

• Figure 1: Percentages of users per political coalition in the real-world data and agents in the simulations. For the simulations, each bar represents the mean value across simulation runs with confidence $0.95$.
• Figure 2: Distribution of the Pearson correlations of in-group interactions between the simulation and the real data, divided by model and network initialization strategy. Each dot is a simulation. Overall, approximately 91% of the correlations are not significant ($p>0.05$). Similar results are obtained using Spearman $\rho$.
• Figure 3: Example of comparison between simulated (10 runs across 1 configuration) and real data. On the left, the distribution of in-group interactions percentages for each coalition (using $\texttt{Llama3.2-3B}$, empty initial network, and random recommender system). On the right, the percentages of in-group interactions for each coalition in the real-world dataset.
• Figure 4: Distribution of the Pearson correlations of out-group interactions between the simulation and the real data, divided by model and network initialization strategy. Each dot is a simulation. Overall, approximately 66% of the correlations are not significant ($p>0.05$). Similar results are obtained using Spearman $\rho$.
• Figure 5: Distribution of the Pearson correlations of in-group toxicity between the simulation and the real data, divided by model and network initialization strategy. Each dot is a simulation. Overall, approximately 98% of the correlations are not significant ($p>0.05$). Similar results are obtained using Spearman $\rho$.