Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Risks, Benefits, and Consequences of Prepublication Moderation: Evidence from 17 Wikipedia Language Editions

Chau Tran, Kaylea Champion, Benjamin Mako Hill, Rachel Greenstadt

TL;DR

This study assesses FlaggedRevs, a prepublication moderation system, using data from 17 Wikipedia editions to evaluate its impact on activity, quality, and newcomer retention. Through wiki-level ITS and user-level GLMM analyses, the authors find that prepublication review effectively hides low-quality contributions before publication, but yields only moderate or null effects on overall contribution volume and quality. There is a small, statistically significant but practically modest negative impact on newcomer return rates. The findings suggest that prepublication moderation can reduce harmful content without substantially harming participation, informing moderation design choices across peer-production platforms.

Abstract

Many online communities rely on postpublication moderation where contributors, even those that are perceived as being risky, are allowed to publish material immediately and where moderation takes place after the fact. An alternative arrangement involves moderating content before publication. A range of communities have argued against prepublication moderation by suggesting that it makes contributing less enjoyable for new members and that it will distract established community members with extra moderation work. We present an empirical analysis of the effects of a prepublication moderation system called FlaggedRevs that was deployed by several Wikipedia language editions. We used panel data from 17 large Wikipedia editions to test a series of hypotheses related to the effect of the system on activity levels and contribution quality. We found that the system was very effective at keeping low-quality contributions from ever becoming visible. Although there is some evidence that the system discouraged participation among users without accounts, our analysis suggests that the system's effects on contribution volume and quality were moderate at most. Our findings imply that concerns regarding the major negative effects of prepublication moderation systems on contribution quality and project productivity may be overstated.

The Risks, Benefits, and Consequences of Prepublication Moderation: Evidence from 17 Wikipedia Language Editions

TL;DR

This study assesses FlaggedRevs, a prepublication moderation system, using data from 17 Wikipedia editions to evaluate its impact on activity, quality, and newcomer retention. Through wiki-level ITS and user-level GLMM analyses, the authors find that prepublication review effectively hides low-quality contributions before publication, but yields only moderate or null effects on overall contribution volume and quality. There is a small, statistically significant but practically modest negative impact on newcomer return rates. The findings suggest that prepublication moderation can reduce harmful content without substantially harming participation, informing moderation design choices across peer-production platforms.

Abstract

Many online communities rely on postpublication moderation where contributors, even those that are perceived as being risky, are allowed to publish material immediately and where moderation takes place after the fact. An alternative arrangement involves moderating content before publication. A range of communities have argued against prepublication moderation by suggesting that it makes contributing less enjoyable for new members and that it will distract established community members with extra moderation work. We present an empirical analysis of the effects of a prepublication moderation system called FlaggedRevs that was deployed by several Wikipedia language editions. We used panel data from 17 large Wikipedia editions to test a series of hypotheses related to the effect of the system on activity levels and contribution quality. We found that the system was very effective at keeping low-quality contributions from ever becoming visible. Although there is some evidence that the system discouraged participation among users without accounts, our analysis suggests that the system's effects on contribution volume and quality were moderate at most. Our findings imply that concerns regarding the major negative effects of prepublication moderation systems on contribution quality and project productivity may be overstated.
Paper Structure (15 sections, 2 equations, 5 figures, 5 tables)

This paper contains 15 sections, 2 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Peer Production and Barriers to Entry
  4. Moderation and Community Health
  5. Moderation Design Decisions
  6. Testing the Effects of Prepublication Moderation on Contribution Actions
  7. Empirical Setting
  8. Methods
  9. Data
  10. Outcome Measures
  11. Analytical Plan
  12. Results
  13. Discussion
  14. Limitations
  15. Conclusion

Figures (5)

  • Figure 1: Impact of FlaggedRevs on the quantity of visible reverted contributions (in standard deviation units) made by different groups of users. The vertical line indicates the start of the intervention. The regression lines along with confidence bands are fitted using a linear model.
  • Figure 2: Impact of FlaggedRevs on the number of reverted contributions (in standard deviation units) made by different groups of users.
  • Figure 3: Impact of FlaggedRevs on the reverted rate (in standard deviation units) of contributions made by different groups of users.
  • Figure 4: Impact of FlaggedRevs on the number of contributions (in standard deviation units) among different groups of users.
  • Figure 5: Predicted return rate from our model four prototypical newcomers from our model testing H4. These predicted values help interpret the three-way interaction between different independent variables on newcomer return rate on wikis that implemented FlaggedRevs.