Did the Roll-Out of Community Notes Reduce Engagement With Misinformation on X/Twitter?

TL;DR

The paper investigates whether X/Twitter's crowdsourced Community Notes reduces engagement with misinformation by applying Difference-in-Differences and Regression Discontinuity designs to a large, real-world dataset spanning the pilot and global rollout periods. Despite a substantial increase in fact-checking activity and faster note creation, the study finds no robust evidence that Community Notes significantly decreases engagement metrics such as $RetweetCount$ and $LikeCount$; observed declines appear to reflect broader platform dynamics rather than targeted effects. The results underscore the importance of field evaluations for misinformation interventions and suggest that note display delays—often longer than a tweet's diffusion window—limit impact in early viral stages. The findings have practical implications for improving crowdsourced fact-checking by prioritizing speed, preselection of high-risk content, and integration with broader media-literacy efforts.

Abstract

Developing interventions that successfully reduce engagement with misinformation on social media is challenging. One intervention that has recently gained great attention is X/Twitter's Community Notes (previously known as "Birdwatch"). Community Notes is a crowdsourced fact-checking approach that allows users to write textual notes to inform others about potentially misleading posts on X/Twitter. Yet, empirical evidence regarding its effectiveness in reducing engagement with misinformation on social media is missing. In this paper, we perform a large-scale empirical study to analyze whether the introduction of the Community Notes feature and its roll-out to users in the U.S. and around the world have reduced engagement with misinformation on X/Twitter in terms of retweet volume and likes. We employ Difference-in-Differences (DiD) models and Regression Discontinuity Design (RDD) to analyze a comprehensive dataset consisting of all fact-checking notes and corresponding source tweets since the launch of Community Notes in early 2021. Although we observe a significant increase in the volume of fact-checks carried out via Community Notes, particularly for tweets from verified users with many followers, we find no evidence that the introduction of Community Notes significantly reduced engagement with misleading tweets on X/Twitter. Rather, our findings suggest that Community Notes might be too slow to effectively reduce engagement with misinformation in the early (and most viral) stage of diffusion. Our work emphasizes the importance of evaluating fact-checking interventions in the field and offers important implications to enhance crowdsourced fact-checking strategies on social media.

Figures (9)

• Figure 1: Daily counts of created notes and fact-checked tweets, smoothed by a 7-day moving average. There are two grey vertical dash lines. The first (left) is on October 6, 2022 (Community Notes feature visible in the U. S.), and the second (right) is on December 11, 2022 (Community Notes feature visible to the world).
• Figure 2: Complementary cumulative distribution functions (CCDFs) of the delays. (a) Delay (response time) between the time of tweet creation and the time of note creation. (b) Delay (response time) between the time of tweet creation and the time when the note is displayed to users.
• Figure 3: Two-week rolling averages of retweet count and like count in treatment (F-CRH) and control (F-NCRH) groups from 2022-07 to 2023-04. The two vertical grey dash lines in each figure indicate two dates, October 6, 2022 (left) and December 11, 2022 (right), when Community Notes was expanded to the U. S. and the world respectively. (a) The points are 2-week rolling averages of retweet count. The horizontal dash lines represent the averages of retweet count during different time periods, and illustrate the results of DiD models in Columns (1) and (2) of Table \ref{['tab:mis_all_did']}. The lighter red dash line during $\mathit{VisUS}$ denotes the estimated average without the treatment effect ($\mathit{FCRH \times Vis} = -0.439, p < 0.01$) in Column (1) of Table \ref{['tab:mis_all_did']}. (b) The points are 2-week rolling averages of like count. The horizontal dash lines represent the averages of like count during different time periods, and illustrate the results of DiD models in Columns (3) and (4) of Table \ref{['tab:mis_all_did']}. The lighter red dash line during $\mathit{VisUS}$ denotes the estimated average without the treatment effect ($\mathit{FCRH \times Vis} = -0.403, p < 0.05$) in Column (3) of Table \ref{['tab:mis_all_did']}.
• Figure 4: Two-week rolling averages of retweet count and like count in treatment (F-CRH) and control (T-NMR) groups from 2022-07 to 2023-04. The two vertical grey dash lines in each figure indicate two dates, October 6, 2022 (left) and December 11, 2022 (right), when Community Notes was expanded to the U. S. and the world respectively. (a) The points are 2-week rolling averages of retweet count. The horizontal dash lines represent the averages of retweet count during different time periods, and illustrate the results of DiD models in Columns (1) and (2) of Table \ref{['tab:tf_high_rank_did']}. The two lines are always parallel over time, which corresponds to the statistically non-significant coefficient estimates of $\mathit{FCRH \times Vis}$ in Columns (1) and (2). (b) The points are 2-week rolling averages of like count. The horizontal dash lines represent the averages of like count during different time periods, and illustrate the results of DiD models in Columns (3) and (4) of Table \ref{['tab:tf_high_rank_did']}. The two lines are always parallel over time, which corresponds to the statistically non-significant coefficient estimates of $\mathit{FCRH \times Vis}$ in Columns (3) and (4).
• Figure 5: The averages of retweet count and like count across the note score points. Note scores are re-centered based on the cutoff point of 0.40. The shadow areas represent 95% Confidence Intervals. (a) The averages of retweet count across the note score points. There is no obvious discontinuity on the cutoff point. (b) The averages of like count across the note score points. There is no obvious discontinuity on the cutoff point.