Venire: A Machine Learning-Guided Panel Review System for Community Content Moderation

TL;DR

Venire is developed, an ML-backed system for panel review on Reddit that represents a novel paradigm for human-AI content moderation, and shifts the conversation from replacing human decision-making to supporting it.

Abstract

Research into community content moderation often assumes that moderation teams govern with a single, unified voice. However, recent work has found that moderators disagree with one another at modest, but concerning rates. The problem is not the root disagreements themselves. Subjectivity in moderation is unavoidable, and there are clear benefits to including diverse perspectives within a moderation team. Instead, the crux of the issue is that, due to resource constraints, moderation decisions end up being made by individual decision-makers. The result is decision-making that is inconsistent, which is frustrating for community members. To address this, we develop Venire, an ML-backed system for panel review on Reddit. Venire uses a machine learning model trained on log data to identify the cases where moderators are most likely to disagree. Venire fast-tracks these cases for multi-person review. Ideally, Venire allows moderators to surface and resolve disagreements that would have otherwise gone unnoticed. We conduct three studies through which we design and evaluate Venire: a set of formative interviews with moderators, technical evaluations on two datasets, and a think-aloud study in which moderators used Venire to make decisions on real moderation cases. Quantitatively, we demonstrate that Venire is able to improve decision consistency and surface latent disagreements. Qualitatively, we find that Venire helps moderators resolve difficult moderation cases more confidently. Venire represents a novel paradigm for human-AI content moderation, and shifts the conversation from replacing human decision-making to supporting it.

Figures (12)

• Figure 1: Venire workflow. A case is pulled from the moderation queue by a human moderator, and an AI model recommends whether it should be reviewed by a single moderator or a panel of moderators.
• Figure 2: Two potential Venire workflows. Left: The strict voting workflow. When making a ruling on a case, moderators must specify whether they want to flag it for panel review or not. Cases flagged for panel review remain in the moderation queue until a majority vote is achieved across $k$ raters. Right: The suggested action workflow. Rather than enforcing a strict voting procedure, moderators are always given the option to "suggest" an action instead of making a decision, making their opinion visible to other moderators. Any moderator can input a final decision when they feel confident enough.
• Figure 3: Visualization of how the model predicts the moderation team will react to a particular case---accessed by clicking the "More info" button next to the panel recommendation text in both mock-ups.
• Figure 4: Overview of our model architecture. We prepend a unique token representing a specific annotator (a) to each sentence, and pass the finetuned BERT contextual embedding for this token through a fully connected layer (b) to produce our final prediction. We ignore the special CLS token that is typically used for BERT sequence classification tasks devlin2018bert.
• Figure 5: Performance of a panel prioritization strategies at increasing workloads. When it comes to improving decision consistency (RQ2a), majority-vote based prioritization converges to the optimal value much faster than random allocation. Majority-vote based prioritization also outperforms other strategies at surfacing disagreements(RQ2b), though to a less dramatic degree