AI Feedback Enhances Community-Based Content Moderation through Engagement with Counterarguments

TL;DR

An AI-assisted hybrid moderation framework in which participants receive AI-generated feedback, supportive, neutral, or argumentative, on their notes and are asked to revise them accordingly is explored, showing that incorporating feedback improves the quality of notes.

Abstract

Today, social media platforms are significant sources of news and political communication, but their role in spreading misinformation has raised significant concerns. In response, these platforms have implemented various content moderation strategies. One such method, Community Notes (formerly Birdwatch) on X (formerly Twitter), relies on crowdsourced fact-checking and has gained traction. However, it faces challenges such as partisan bias and delays in verification. This study explores an AI-assisted hybrid moderation framework in which participants receive AI-generated feedback, supportive, neutral, or argumentative, on their notes and are asked to revise them accordingly. The results show that incorporating feedback improves the quality of notes, with the most substantial gains resulting from argumentative feedback. This underscores the value of diverse perspectives and direct engagement in human-AI collective intelligence. The research contributes to ongoing discussions about AI's role in political content moderation, highlighting the potential of generative AI and the importance of informed design.

Figures (18)

• Figure 1: The experimental workflow illustrates the process of note creation, feedback assignment, and evaluation. Participants (self-identified as Democrats or Republicans) wrote initial notes to provide context on posts authored by either Democrats or Republicans. They then received randomly assigned feedback varying in type (supportive, neutral, or argumentative) and source label (AI agent or human expert). After revising their notes in response to the feedback, a separate group of self-identified Democrats and Republicans rated the helpfulness of the original and revised notes. This design enables analysis of feedback effects on note quality and partisan differences in evaluation.
• Figure 2: Examples of notes with high and low engagement in response to argumentative feedback. (a) A note that improved following argumentative feedback. (b) A note that declined following argumentative feedback
• Figure 3: Logistic regression for the note rating improvement. Coefficient estimates from the logistic regression model for note rating improvement as a function of $FA$. Points represent estimated log-odds, with horizontal bars indicating 95% confidence intervals. A vertical dashed line at 0 denotes the null effect. Estimates are grouped by rater affiliation, with blue indicating Democrats and red indicating Republicans.
• Figure 4: Bar plot of OLS model for $FA$. (a) Model 1 includes feedback type (Neutral as the reference), post type (Republican as the reference), and participant partisanship (Republican as the reference). (b) Model 2 includes feedback type, the alignment between participant and post-partisanship (Co- vs. Cross-partisan), and their interaction, with Cross-partisan × Neutral as the reference category. The plots show estimated regression coefficients ($\beta$) with one-standard-error bars. The vertical dashed line at 0 indicates the absence of an effect. Bar widths reflect the relative magnitude of coefficients, and fill colours indicate statistical significance (two-sided Wald tests).
• Figure 5: Comparison of $FA$ values by treatment condition and feedback source for different source labels. Mean $FA$ values are shown for each feedback type and the alignment between the participant and the post partisanship (Co- vs. Cross-partisan), with bars coloured by the feedback source (AI vs. Human). Error bars represent the standard error of the mean.