Wikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia

TL;DR

Wikibench tackles the problem that AI evaluation datasets are often curated by outside developers, which can misalign with community norms. The authors introduce Wikibench, a system integrated into Wikipedia that enables community members to select, label, and discuss data points to build evaluation datasets reflecting consensus, disagreement, and uncertainty. A field study with Wikipedians demonstrates that the resulting labels better reflect community consensus than outsider-labeling and that the process empowers participants to shape higher-level data-curation decisions. The work highlights the potential for community-driven AI evaluation to improve alignment with diverse values and outlines directions for extending such approaches to other communities and contexts.

Abstract

AI tools are increasingly deployed in community contexts. However, datasets used to evaluate AI are typically created by developers and annotators outside a given community, which can yield misleading conclusions about AI performance. How might we empower communities to drive the intentional design and curation of evaluation datasets for AI that impacts them? We investigate this question on Wikipedia, an online community with multiple AI-based content moderation tools deployed. We introduce Wikibench, a system that enables communities to collaboratively curate AI evaluation datasets, while navigating ambiguities and differences in perspective through discussion. A field study on Wikipedia shows that datasets curated using Wikibench can effectively capture community consensus, disagreement, and uncertainty. Furthermore, study participants used Wikibench to shape the overall data curation process, including refining label definitions, determining data inclusion criteria, and authoring data statements. Based on our findings, we propose future directions for systems that support community-driven data curation.

Figures (13)

• Figure 1: Wikibench's workflow. Wikibench mainly supports three actions for community-driven data curation: select, label, and discuss, each illustrated by different colors. Community members can select and label data points during their regular activities (i.e., while patrolling for damaging edits on Wikipedia) or choose from data points already collected in the dataset. They can also discuss individual data points to resolve disagreements or initiate a higher-level discussion related to the overall data curation process.
• Figure 2: Wikibench's plug-in is embedded in Wikipedia's diff pages, where Wikipedians already assess edits during their regular patrolling activities. Through the plug-in, Wikipedians can label an edit's damage and user intent, specify their confidence level, and add notes if desired.
• Figure 3: The message displayed after Wikipedians successfully submit their labels using the plug-in. The yellow message appears only when the submitted labels differ from the current primary label to facilitate discussion.
• Figure 4: Wikibench's entity page for a given edit. The left side shows the top half of an entity page, featuring the edit, its primary label, and the user's individual label. The right side shows the bottom half of an entity page, containing the full set of individual labels and accompanying notes. Participants' usernames are blurred to avoid identification.
• Figure 5: The message displayed when Wikipedians edit primary labels to encourage them to be bold yet respectful of others' views.