Assessing the Potential of Generative Agents in Crowdsourced Fact-Checking

TL;DR

This paper investigates the feasibility of using generative agents powered by large language models to participate in crowdsourced fact-checking. It adopts the experimental protocol of a prior human crowdsourcing study, building a synthetic crowd with demographically diverse agent profiles and a two-phase workflow: evidence selection and structured questionnaire-based evaluation. Across 70 statements, the agent crowds (three different LLM backbones) generally outperform human crowds in truthfulness classification and exhibit higher internal consistency and bias resilience, particularly when external evidence is available. The findings support the potential of generative agents as scalable, consistent, and less biased contributors to crowd-based misinformation mitigation, while also highlighting limitations related to data scope, domain variability, and ethical implications.

Abstract

The growing spread of online misinformation has created an urgent need for scalable, reliable fact-checking solutions. Crowdsourced fact-checking - where non-experts evaluate claim veracity - offers a cost-effective alternative to expert verification, despite concerns about variability in quality and bias. Encouraged by promising results in certain contexts, major platforms such as X (formerly Twitter), Facebook, and Instagram have begun shifting from centralized moderation to decentralized, crowd-based approaches. In parallel, advances in Large Language Models (LLMs) have shown strong performance across core fact-checking tasks, including claim detection and evidence evaluation. However, their potential role in crowdsourced workflows remains unexplored. This paper investigates whether LLM-powered generative agents - autonomous entities that emulate human behavior and decision-making - can meaningfully contribute to fact-checking tasks traditionally reserved for human crowds. Using the protocol of La Barbera et al. (2024), we simulate crowds of generative agents with diverse demographic and ideological profiles. Agents retrieve evidence, assess claims along multiple quality dimensions, and issue final veracity judgments. Our results show that agent crowds outperform human crowds in truthfulness classification, exhibit higher internal consistency, and show reduced susceptibility to social and cognitive biases. Compared to humans, agents rely more systematically on informative criteria such as Accuracy, Precision, and Informativeness, suggesting a more structured decision-making process. Overall, our findings highlight the potential of generative agents as scalable, consistent, and less biased contributors to crowd-based fact-checking systems.

Figures (3)

• Figure 1: Framework Overview. The framework consists of two main phases: data preparation and simulation. During the data preparation phase, the dataset is tailored for generative agents, involving statement selection, evidence curation, and agent profile design to replicate human crowd diversity. The simulation phase mirrors a crowd-based fact-checking process, where generative agents perform two primary tasks: selecting the most relevant evidence and completing a structured questionnaire to assess the truthfulness and quality dimensions of statements.
• Figure 2: Distribution of ratings assigned by generative agents and human workers across different truthfulness levels. The boxplots highlight the consistency of agent ratings, particularly for extreme labels. In contrast, human workers exhibit wider interquartile ranges and more outliers, indicating higher inconsistency in their assessments.
• Figure 3: Average ratings assigned by generative agents and human evaluators to each quality dimension across different truthfulness levels. Generative agents consistently rated "Comprehensibility" higher and were more stringent in "Completeness", with ratings rising alongside truthfulness. Human evaluators showed greater variability and gave lower scores for less truthful statements. Furthermore, agents demonstrated greater sensitivity to truthfulness in dimensions like "Accuracy" and "Precision". while human ratings were more moderate and less polarized.