Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Threats to the sustainability of Community Notes on X

Zahra Arjmandi-Lari, Alexios Mantzarlis, Tom Stafford

TL;DR

This paper investigates the sustainability of X's Community Notes by analyzing contributor behavior and the causal impact of note publication. It employs a regression discontinuity design around the publication threshold of $0.4$ to estimate how having a note published affects future note-writing, revealing a small but robust $"$ $5\%$ increase in subsequent contributions$"$ for near-threshold authors. Key findings show rapid growth and high concentration of activity among a small subset of power users, substantial churn among authors, and a large share of notes remaining unseen due to ratings diversity requirements, underscoring systemic sustainability risks. The work also discusses implications for broader adoption of crowd-sourced moderation and the potential effects of AI-generated notes on human participation.

Abstract

Community Notes are emerging as an important option for content moderation. The Community Notes system pioneered by Twitter, now known as X, uses a bridging algorithm to identify user-generated context with upvotes across political divides, supposedly spinning consensual gold from partisan straw. It is important to understand the nature of the community behind Community Notes, especially as the feature has now been imitated by several billion-user platforms. We look for signs of stability and disruption in the X Community Notes community and interrogate the motivations other than partisan animus (Allen, Martel, and Rand 2022) which may be driving users to contribute. We conduct a novel analysis of the impact of having a note published, which requires being considered "helpful" by the bridging algorithm, utilising a regression discontinuity design. This allows stronger causal inference than conventional methods used with observational data. Our analysis shows the positive effect on future note authoring of having a note published. This highlights the risk of the current system, where the proportion of notes considered "helpful" (and therefore shown to users on X) is low, 10%, and declining. This analysis has implications for the future of Community Notes on X and the extension of this approach to other platforms.

Threats to the sustainability of Community Notes on X

TL;DR

This paper investigates the sustainability of X's Community Notes by analyzing contributor behavior and the causal impact of note publication. It employs a regression discontinuity design around the publication threshold of to estimate how having a note published affects future note-writing, revealing a small but robust increase in subsequent contributions for near-threshold authors. Key findings show rapid growth and high concentration of activity among a small subset of power users, substantial churn among authors, and a large share of notes remaining unseen due to ratings diversity requirements, underscoring systemic sustainability risks. The work also discusses implications for broader adoption of crowd-sourced moderation and the potential effects of AI-generated notes on human participation.

Abstract

Community Notes are emerging as an important option for content moderation. The Community Notes system pioneered by Twitter, now known as X, uses a bridging algorithm to identify user-generated context with upvotes across political divides, supposedly spinning consensual gold from partisan straw. It is important to understand the nature of the community behind Community Notes, especially as the feature has now been imitated by several billion-user platforms. We look for signs of stability and disruption in the X Community Notes community and interrogate the motivations other than partisan animus (Allen, Martel, and Rand 2022) which may be driving users to contribute. We conduct a novel analysis of the impact of having a note published, which requires being considered "helpful" by the bridging algorithm, utilising a regression discontinuity design. This allows stronger causal inference than conventional methods used with observational data. Our analysis shows the positive effect on future note authoring of having a note published. This highlights the risk of the current system, where the proportion of notes considered "helpful" (and therefore shown to users on X) is low, 10%, and declining. This analysis has implications for the future of Community Notes on X and the extension of this approach to other platforms.

Paper Structure

This paper contains 9 sections, 7 figures.

Table of Contents

  1. Introduction
  2. Prior work
  3. Data collection and methods
  4. Results
  5. A growing community driven by a subset of power users
  6. Community Notes authors persist and get replaced, but are not all equal
  7. Most notes are unseen, which may have a long-term effect
  8. Discussion
  9. Acknowledgments

Figures (7)

  • Figure 1: Monthly Active Authors (MAAs) contributed at least one note during the reference month
  • Figure 2: Share of helpful notes authored by percentile of authors in 2024. Dashed line shows median (50th percentile) point.
  • Figure 3: Persistence rate of authors based on the year of the first note written.
  • Figure 4: Fraction of authors who stayed active (defined as authoring another note within 4 months of their first note).
  • Figure 5: Note all-time highest helpfulness score according to the standard algorithm (“Max core note intercept”) against log Number of user ratings, colored by current note status (rated helpful: green; rated not helpful: red; rated “needs more rating”: orange), for 2024 notes.
  • ...and 2 more figures