References to unbiased sources increase the helpfulness of community fact-checks

TL;DR

It is found that community-created fact-checks linking to high-bias sources (of either political side) are perceived as significantly less helpful, which suggests that the rating mechanism on the Community Notes platform successfully penalizes one-sidedness and politically motivated reasoning.

Abstract

Community-based fact-checking is a promising approach to address misinformation on social media at scale. However, an understanding of what makes community-created fact-checks helpful to users is still in its infancy. In this paper, we analyze the determinants of the helpfulness of community-created fact-checks. For this purpose, we draw upon a unique dataset of real-world community-created fact-checks and helpfulness ratings from X's (formerly Twitter) Community Notes platform. Our empirical analysis implies that the key determinant of helpfulness in community-based fact-checking is whether users provide links to external sources to underpin their assertions. On average, the odds for community-created fact-checks to be perceived as helpful are 2.70 times higher if they provide links to external sources. Furthermore, we demonstrate that the helpfulness of community-created fact-checks varies depending on their level of political bias. Here, we find that community-created fact-checks linking to high-bias sources (of either political side) are perceived as significantly less helpful. This suggests that the rating mechanism on the Community Notes platform successfully penalizes one-sidedness and politically motivated reasoning. These findings have important implications for social media platforms, which can utilize our results to optimize their community-based fact-checking systems.

Figures (5)

• Figure 1: Example of a Community-Created Fact-Check ("Community Note") on X.
• Figure 2: Distribution of political biases in Community Notes. (A) Bias magnitude ordered from Low to High. (B) Bias direction, separated into Left, Undirected, and Right.
• Figure 3: Binomial regression analyzing the helpfulness of external sources in explaining the share of helpful votes. Shown are coefficient estimates with 95% CIs. Unit of analysis is the fact-check level ($N = 41,129$).
• Figure 4: Marginal effects (with 95% CIs) of bias magnitude (left panel) and bias direction (right panel) on the share of helpful votes. Unit of analysis is the fact-check level ($N = 21,307$).
• Figure 5: Analysis across media types. (A) Results of a binomial regression analyzing the helpfulness of external sources varies across different media types. Shown are coefficient estimates with 95% CIs. Unit of analysis is the fact-check level ($N = 21,277$). (B) Frequencies of media types across community notes.