Wisdom of the LLM Crowd: A Large Scale Benchmark of Multi-Label U.S. Election-Related Harmful Social Media Content

TL;DR

The paper addresses the challenge of scalable, nuanced labeling of election-related harmful content on social media by introducing USE24-XD, a near 100k-post dataset collected from X during the 2024 U.S. election window. It evaluates six LLMs (two per family) in a zero-shot, multi-label framework across five categories and uses wisdom-of-the-crowd via majority voting among top models, validated against 34 MTurk annotators and human consensus labels. Inter-rater reliability analyses employing Cohen’s $\kappa$ and Krippendorff’s $\alpha$ show that LLMs achieve competitive agreement and, when ensembled, can match or exceed human reliability while reducing cost, with recall up to $0.90$ for Speculation and strong performance on Conspiracy. The dataset is publicly released to spur future research, and the study highlights how annotator demographics shape labeling patterns, underscoring the need to consider human factors alongside model calibration in large-scale content analysis.

Abstract

The spread of election misinformation and harmful political content conveys misleading narratives and poses a serious threat to democratic integrity. Detecting harmful content at early stages is essential for understanding and potentially mitigating its downstream spread. In this study, we introduce USE24-XD, a large-scale dataset of nearly 100k posts collected from X (formerly Twitter) during the 2024 U.S. presidential election cycle, enriched with spatio-temporal metadata. To substantially reduce the cost of manual annotation while enabling scalable categorization, we employ six large language models (LLMs) to systematically annotate posts across five nuanced categories: Conspiracy, Sensationalism, Hate Speech, Speculation, and Satire. We validate LLM annotations with crowdsourcing (n = 34) and benchmark them against human annotators. Inter-rater reliability analyses show comparable agreement patterns between LLMs and humans, with LLMs exhibiting higher internal consistency and achieving up to 0.90 recall on Speculation. We apply a wisdom-of-the-crowd approach across LLMs to aggregate annotations and curate a robust multi-label dataset. 60% of posts receive at least one label. We further analyze how human annotator demographics, including political ideology and affiliation, shape labeling behavior, highlighting systematic sources of subjectivity in judgments of harmful content. The USE24-XD dataset is publicly released to support future research.

Figures (15)

• Figure 1: Overall Method for Multi-label Harmful Online Content Annotation
• Figure 2: Distribution of each category annotated by different LLM and crowdsourced workers (majority vote). Each category was evaluated independently.
• Figure 3: Strength of association between annotators’ demographics and annotation categories, expressed as Cramér's V from Pearson's $\chi^{2}$ tests ($N\approx3{,}000$ assignments). Darker cells indicate stronger associations.
• Figure 4: Krippendorff’s Alpha ($\alpha$) Comparison. (a) shows $\alpha$ computed for all combinations of three LLMs selected from six models across the full dataset. (b) presents $\alpha$ for 193 groups of human annotators on the subset of data where annotations were available. The values range from -1 to 1, where higher $\alpha$ values indicate stronger agreement.
• Figure 5: Direct comparison between majority-vote labels produced by LLM ensembles (20 outputs) and majority-vote labels from crowdsourced workers across different categories.