Beyond the "Truth": Investigating Election Rumors on Truth Social During the 2024 Election

TL;DR

This work tackles the measurement of election rumor dynamics on Truth Social using a multi-stage Rumor Detection Agent that fuses a fine-tuned RoBERTa classifier, keyword filters, and a two-pass LLM verifier (GPT-4o mini) to label rumors at scale. It builds a dataset of nearly 15 million posts from about 200,000 users and demonstrates a dose-response illusory truth effect, showing that repeated exposure markedly increases resharing propensity. Network diffusion simulations reveal rapid contagion, with up to roughly 14–25% of users becoming infected within a few propagation steps, and identify Donald Trump as a central node driving major surges in rumor spread. The study illustrates the potential of LLMs for psychological measurement in large real-world data, and suggests practical pathways for early detection and prebunking interventions to mitigate misinformation on niche, ideologically homogeneous platforms.

Abstract

Large language models (LLMs) offer unprecedented opportunities for analyzing social phenomena at scale. This paper demonstrates the value of LLMs in psychological measurement by (1) compiling the first large-scale dataset of election rumors on a niche alt-tech platform, (2) developing a multistage Rumor Detection Agent that leverages LLMs for high-precision content classification, and (3) quantifying the psychological dynamics of rumor propagation, specifically the "illusory truth effect" in a naturalistic setting. The Rumor Detection Agent combines (i) a synthetic data-augmented, fine-tuned RoBERTa classifier, (ii) precision keyword filtering, and (iii) a two-pass LLM verification pipeline using GPT-4o mini. The findings reveal that sharing probability rises steadily with each additional exposure, providing large-scale empirical evidence for dose-response belief reinforcement in ideologically homogeneous networks. Simulation results further demonstrate rapid contagion effects: nearly one quarter of users become "infected" within just four propagation iterations. Taken together, these results illustrate how LLMs can transform psychological science by enabling the rigorous measurement of belief dynamics and misinformation spread in massive, real-world datasets.

Figures (15)

• Figure 1: Webscraping user metadata and posts from Truth Social's website. The database schema is detailed in Figure S1.
• Figure 2: Rumor Detection Agent used to classify Truth Social posts. The system uses a fine-tuned RoBERTa model and keyword filtering to identify posts with high probabilities of being a rumor. The posts are then sent to an LLM (GPT-4o mini) where few shot prompting is used first to classify the post as rumor. A second prompt then ensures that the post corresponds to at least one specific rumor.
• Figure 3: Temporal breakdown of rumor posts by rumor type during the key election window.
• Figure 4: Cumulative sharing probability $P^{\text{share}}(k)$ as a function of the number of exposures $k$. Blue line shows sharing probability; grey bars show sample size at each exposure level.
• Figure 5: Election margin vs. rumor rate by state. Trend line shows positive correlation ($r = 0.34$).