The Benefit of Collective Intelligence in Community-Based Content Moderation is Limited by Overt Political Signalling

TL;DR

It is hypothesized that enabling users to collaborate on writing notes, rather than relying solely on individually authored notes, can enhance their overall quality, and that politically diverse teams perform better when evaluating Republican posts, while group composition does not affect perceived note quality for Democrat posts.

Abstract

Social media platforms face increasing scrutiny over the rapid spread of misinformation. In response, many have adopted community-based content moderation systems, including Community Notes (formerly Birdwatch) on X (formerly Twitter), Footnotes on TikTok, and Facebook's Community Notes initiative. However, research shows that the current design of these systems can allow political biases to influence both the development of notes and the rating processes, reducing their overall effectiveness. We hypothesize that enabling users to collaborate on writing notes, rather than relying solely on individually authored notes, can enhance their overall quality. To test this idea, we conducted an online experiment in which participants jointly authored notes on political posts. Our results show that teams produce notes that are rated as more helpful than individually written notes. We also find that politically diverse teams perform better when evaluating Republican posts, while group composition does not affect perceived note quality for Democrat posts. However, the advantage of collaboration diminishes when team members are aware of one another's political affiliations. Taken together, these findings underscore the complexity of community-based content moderation and highlight the importance of understanding group dynamics and political diversity when designing more effective moderation systems.

Figures (20)

• Figure 1: Experimental design. Participants wrote individual and collaborative notes for 40 political posts sourced from Democratic and Republican accounts. We randomly paired participants with partners who either shared or differed in political affiliation, forming three group types: Democrat–Democrat (DD), Republican–Republican (RR), and Democrat–Republican (DR). Each team was then randomly assigned to one of two treatments: the overt condition, in which participants could see each other's political affiliations, or the covert condition, in which affiliations were not disclosed.
• Figure 2: Distribution of helpfulness scores according to experts. The distribution of helpfulness scores according to expert ratings, $H_{\rm E}$, for individually written notes and collaboratively written notes. (** indicates $p < 0.01$)).
• Figure 3: Distribution of ${I}_{\rm{E}}$ of notes categorised by group's political composition (a) Distributions of ${I}_{\rm E}$ of Democratic posts across groups with different political compositions: Democrats teams (DD), diverse teams (DR), and Republican teams (RR). (b) Distributions of ${I}_{\rm E}$ of Republican posts across groups with different political compositions: Republican teams (RR), diverse teams (DR), and Democrat teams (DD) (* indicates $p < 0.05$).
• Figure 4: Distribution of the change in helpfulness for collaborative notes under Covert and Overt treatments, based on both crowd-sourced and expert assessments. (a) The distribution of ${I}_{\rm E}$ (b) The distribution of $I_{\rm{D}}$ (c) The distribution of $I_{\rm{R}}$ for covert and overt treatments. ** indicates $p < 0.01$)
• Figure S1: Individual evaluation task interface. Participants reviewed a social media post and judged its credibility. They then provided a written explanation of up to 280 characters, with optional contextual details and source citations (URLs excluded from the character limit), following instructions modelled on Community Notes. Participants had up to seven minutes to complete the task before an attention check was triggered; those who passed were redirected to the collaborative evaluation phase.