From Birdwatch to Community Notes, from Twitter to X: four years of community-based content moderation

TL;DR

A systematic review of the literature on Community Notes, and a major curated dataset and accompanying source code to support future research on Community Notes are provided.

Abstract

Community Notes (formerly known as Birdwatch) is the first large-scale crowdsourced content moderation initiative that was launched by X (formerly known as Twitter) in January 2021. As the Community Notes model gains momentum across other social media platforms, there is a growing need to assess its underlying dynamics and effectiveness. This Resource paper provides (a) a systematic review of the literature on Community Notes, and (b) a major curated dataset and accompanying source code to support future research on Community Notes. We parsed Notes and Ratings data from the first four years of the program and conducted language detection across all Notes. Focusing on English-language Notes, we extracted embedded URLs and identified discussion topics in each Note. Additionally, we constructed monthly interaction networks among the Contributors. Together with the literature review, these resources offer a robust foundation for advancing research on the Community Notes system.

Figures (8)

• Figure 1: Examples of Notes. (a) A Note that requires more ratings and is shown only to Community Notes Contributors for additional rating. (b) A Note that has received the status of helpful and is publicly shown beneath a Post.
• Figure 2: Classification and the number of Notes per month. (a) The number of Notes and Ratings created within each month on a log scale. (b) The monthly number of Notes that classify the Posts as "potentially misleading or misinformed" and "not misleading" over four years on a log scale.
• Figure 3: The distribution of Notes and Ratings of Contributors.(a) Log-log rank-plot showing the number of Notes written by each Contributor vs their rank. (b) Log-log rank-plot of the number of Ratings committed by each Contributor vs their rank.
• Figure 4: The distribution of Notes and Ratings of Posts. (a) Log-log rank plot of the number of Notes written on each Post. (b) Log-log rank plot of the number of Ratings each Note has received. (c) Log-log rank plot of the total number of Ratings on all Notes written on each Post. (d) Scatter plot (log-log) of the total number of Ratings on all Notes written on each Post versus the number of Notes written on that Post. The red dots represent the binned mean number of Ratings for a given range in the number of Notes. The black dashed line shows the fitted power-law relationship $y=A\cdot x^b$, where $A = 46$ is the scaling coefficient and $b = 1.74$ is the exponent.
• Figure 5: Top 10 languages in Community Notes. The number of Notes written in each of the top 10 languages used in Community Notes.