Timeliness, Consensus, and Composition of the Crowd: Community Notes on X
Olesya Razuvayevskaya, Adel Tayebi, Ulrikke Dybdal Sørensen, Kalina Bontcheva, Richard Rogers
TL;DR
The paper provides the first large-scale quantitative evaluation of X's Community Notes, identifying a highly skewed contributor landscape, persistent disagreement, and substantial delays in moderation. By combining inequality metrics, dual consensus definitions, and time-lag analyses on a dataset of approximately 1.8 million notes, it shows that CNs are dominated by a small elite, with broad dissensus and timeliness constraints that undermine proactive moderation. The findings suggest that while CNs can surface useful annotations, their effectiveness as a crowdsourced moderation mechanism is limited by participation concentration, algorithmic consensus, and delayed responses, prompting design changes that emphasize deliberation, diversified participation, and performance metrics. The study contributes practical guidance for building more equitable, timely, and epistemically reliable crowd-based moderation systems, and outlines future work on longitudinal patterns and coordinated behaviors.
Abstract
This study presents the first large-scale quantitative analysis of the efficiency of X's Community Notes, a crowdsourced moderation system for identifying and contextualising potentially misleading content. Drawing on over 1.8 million notes, we examine three key dimensions of crowdsourced moderation: participation inequality, consensus formation, and timeliness. Despite the system's goal of collective moderation, we find substantial concentration effect, with the top 10% of contributors producing 58% of all notes (Gini Coefficient = 0.68). The observed consensus is rare-only 11.5% of notes reach agreement on publication, while 69% of posts receive conflicting classifications. A majority of noted posts (approximately 68%) are annotated as "Note Not Needed", reflecting the repurposing of the platform for debate rather than moderation. We found that such posts are paradoxically more likely to yield published notes (OR = 3.12). Temporal analyses show that the notes, on average, are published 65.7 hours after the original post, with longer delays significantly reducing the likelihood of consensus. These results portray Community Notes as a stratified, deliberative system dominated by a small contributor elite, marked by persistent dissensus, and constrained by timeliness. We conclude this study by outlining design strategies to promote equity, faster consensus, and epistemic reliability in community-based moderation.