Supernotes: Driving Consensus in Crowd-Sourced Fact-Checking

TL;DR

This framework uses an LLM to generate many diverse Supernote candidates from existing proposed notes, and shows that both the LLM-based candidate generation and the consensus-driven scoring play crucial roles in creating notes that effectively build consensus among diverse users.

Abstract

X's Community Notes, a crowd-sourced fact-checking system, allows users to annotate potentially misleading posts. Notes rated as helpful by a diverse set of users are prominently displayed below the original post. While demonstrably effective at reducing misinformation's impact when notes are displayed, there is an opportunity for notes to appear on many more posts: for 91% of posts where at least one note is proposed, no notes ultimately achieve sufficient support from diverse users to be shown on the platform. This motivates the development of Supernotes: AI-generated notes that synthesize information from several existing community notes and are written to foster consensus among a diverse set of users. Our framework uses an LLM to generate many diverse Supernote candidates from existing proposed notes. These candidates are then evaluated by a novel scoring model, trained on millions of historical Community Notes ratings, selecting candidates that are most likely to be rated helpful by a diverse set of users. To test our framework, we ran a human subjects experiment in which we asked participants to compare the Supernotes generated by our framework to the best existing community notes for 100 sample posts. We found that participants rated the Supernotes as significantly more helpful, and when asked to choose between the two, preferred the Supernotes 75.2% of the time. Participants also rated the Supernotes more favorably than the best existing notes on quality, clarity, coverage, context, and argumentativeness. Finally, in a follow-up experiment, we asked participants to compare the Supernotes against LLM-generated summaries and found that the participants rated the Supernotes significantly more helpful, demonstrating that both the LLM-based candidate generation and the consensus-driven scoring play crucial roles in creating notes that effectively build consensus among diverse users.

Figures (10)

• Figure 1: An example post, along with three community notes proposed to provide additional context, and the Supernote produced by our framework.
• Figure 2: Overview of our framework for generating Supernotes. (1) We prompt an LLM to generate many candidate Supernotes using the post text and the existing community notes. (2) We score the helpfulness of each candidate by simulating a jury of raters, predicting their ratings, and aggregating them using the Community Notes algorithm. (3) We filter out candidates that do not follow key principles of effective fact-checking. (4) Finally, we rank and select the candidate with the highest score.
• Figure 3: (A) Out-of-sample performance of the classification models predicting whether a rater would rate a note as “helpful”, “somewhat helpful”, or “not helpful” given the post, note, and rater embeddings. (B) Precision of our simulated jury---which aggregates the individual predicted helpfulness ratings---in predicting the most helpful note according to the empirical Community Notes ratings on off-sample historical posts. The dashed vertical lines represent the theoretical baseline of random guessing. The error bars represent 95% CIs.
• Figure 4: Helpfulness ratings. Comparison between the participants’ helpfulness ratings of Supernotes and best existing notes on three-point scale (A) and mapping of the helpfulness ratings to numerical values used in the Community Notes algorithm (B). The Supernotes are rated as significantly more helpful than the best existing notes in both analyses. The error bars represent 95% CIs.
• Figure 5: Helpfulness win rates. The rate at which participants selected the Supernote as more helpful than the best existing community note. Overall, the Supernotes were selected as more helpful 75.2% of the time and 68.9% of the time when both notes were also rated as “helpful”. The error bars represent 95% CIs.