Consensus Stability of Community Notes on X

TL;DR

This paper investigates the stability of Community Notes on X after their initial display, revealing that about $30.2\%$ of notes that are initially rated as helpful later disappear due to post-display polarization in ratings. It combines a reproduced bridging-based note-selection algorithm with a 200-dimensional matrix-factorization model to estimate rater similarity, and employs mixed-effects logistic regression and interrupted time-series analyses to quantify disappearance and rating dynamics. Key findings show sharp post-display increases in rating volume and polarization between raters with similar versus dissimilar viewpoints, with dissimilar raters contributing disproportionately to note disappearance, especially for health, politics, and high-influence posts. The results highlight vulnerabilities in consensus-based moderation under polarization and suggest design and governance approaches, including safeguards, hybrid fact-checking systems, and potential AI-assisted support to improve resilience and stability of community-based fact-checking.

Abstract

Community-based fact-checking systems, such as Community Notes on X (formerly Twitter), aim to mitigate online misinformation by surfacing annotations judged helpful by contributors with diverse viewpoints. While prior work has shown that the platform's bridging-based algorithm effectively selects helpful notes at the time of display, little is known about how evaluations change after notes become visible. Using a large-scale dataset of 437,396 community notes and 35 million ratings from over 580,000 contributors, we examine the stability of helpful notes and the rating dynamics that follow their initial display. We find that 30.2% of displayed notes later lose their helpful status and disappear. Using interrupted time series models, we further show that note display triggers a sharp increase in rating volume and a significant shift in rating leaning, but these effects differ across rater groups. Contributors with viewpoints similar to note authors tend to increase supportive ratings, while dissimilar contributors increase negative ratings, producing systematic post-display polarization. Counterfactual analyses suggest that this post-display polarization, particularly from dissimilar raters, plays a substantial role in note disappearance. These findings highlight the vulnerability of consensus-based fact-checking systems to polarized rating behavior and suggest pathways for improving their resilience.

Figures (7)

• Figure 1: Estimation results for the likelihood of note disappearance. Shown are coefficient estimates with 95% Confidence Intervals (CIs). Random effects at both the author level and the post level are included during estimation.
• Figure 2: Changes in rating count and rating leaning after the initial display of community notes. (a) Average rating count per quarter (15-minute interval) from -16.0 to 16 quarters relative to note display. (b) Average rating leaning per quarter over the same window. The results are shown for all displayed notes, stable notes, and disappeared notes. Error bands represent 95% CIs.
• Figure 3: Discrepancies in rating leaning between similar and dissimilar raters. (a) Average rating leaning per quarter (15-minute interval) for similar raters from -16.0 to 16 quarters relative to note display. (b) Average rating leaning per quarter for general raters over the same window. (c) Average rating leaning per quarter for dissimilar raters over the same window. The results are shown for all displayed notes, stable notes, and disappeared notes. Error bands represent 95% CIs.
• Figure 4: Evaluation of polarized ratings and counterfactual analysis of the note selection algorithm. (a) Estimated changes in rating leaning for similar and dissimilar raters across levels of political bias in external domains cited in community notes. (b) Estimated changes in rating leaning for similar and dissimilar raters across high- and low-quality external domains. (c) Distributions of estimated Core note intercepts (helpfulness) and estimated Core note factors (polarization) from the note selection algorithm, shown for disappeared notes and stable notes. (d) Changes in helpfulness when ratings from similar and dissimilar raters are excluded. (e) Changes in helpfulness when ratings from dissimilar raters are excluded. Results are shown for disappeared notes, stable notes, and notes that have never been displayed. The error bars represent 95% CIs.
• Figure S1: The distribution of estimated Core note intercepts (helpfulness) and estimated Core note factors (polarization) from the note selection algorithm.