Perceptions of Moderators as a Large-Scale Measure of Online Community Governance

TL;DR

This study develops a scalable, data-driven approach to quantify online governance by analyzing how community members publicly discuss their moderators across thousands of Reddit communities. It combines historical moderator timelines, topic and health classification, and a three-stage mod-discourse sentiment classifier, validated for biases and deployed on 8,477 subreddits over 18 months. Using IPTW and DID causal methods, it reveals nuanced, topic- and size-dependent effects of moderation practices on perceived governance, highlighting that engaged, pre/post-tenure moderators and cross-community activity correlate with more positive perceptions, while content removal effects vary by topic and can be polarizing when recruiting publicly. The work offers practical guidance for moderator teams and provides open-source models and anonymized datasets to enable ongoing research in online governance.

Abstract

Millions of online communities are governed by volunteer moderators, who shape their communities by setting and enforcing rules, recruiting additional moderators, and participating in the community themselves. These moderators must regularly make decisions about how to govern, yet measuring the 'success' of governance is complex and nuanced, making it challenging to determine what governance strategies are most successful. Furthermore, prior work has shown that communities have differing values, suggesting that 'one-size-fits-all' approaches to governance are unlikely to serve all communities well. In this work, we assess governance practices on reddit by classifying the sentiment of community members' public discussion of their own moderators. We label 1.89 million posts and comments made on reddit over an 18 month period. We relate these perceptions to characteristics of community governance, and to different actions that community moderators take. We identify types of communities where moderators are perceived particularly positively and negatively, and highlight promising strategies for moderator teams. Amongst other findings, we show that strict rule enforcement is linked to more favorable perceptions of moderators of communities dedicated to certain topics, such as news communities, than others. We investigate what kinds of moderators are associated with improved community perceptions upon their addition to a mod team, and find that moderators who are active community members before and during their mod tenures are seen more favorably. We make our models, anonymized datasets, and code public.

Figures (11)

• Figure 1: Determining the sentiment with regards to the moderators of comments can be very challenging.
• Figure 2: Communities that consider themselves higher quality (a), more trustworthy (b), more engaged (c), more inclusive (d), and more safe (e) all use more positive and less negative sentiment to describe their moderators. Here, communities are grouped into quartiles based on their community members' self-reported perceptions of the current state of the community. This effect is most pronounced for communities' self-reported trustworthiness (b), with the top-25% most trustworthy communities using 34% more positive and 22% less negative language to describe their mods. Communities that rate themselves as feeling smaller (f) have a more positive perception of their mods. In this and all other figures, points represent mean estimates alongside bootstrapped 95% confidence intervals.
• Figure 3: Perceptions of moderators vary significantly across communities with different sizes (a-b) and topics (c-d). In general, smaller communities devote a relatively larger proportion of their content to discussing their moderators (a), and smaller communities express more positive and less negative sentiment towards their mods (b). Discussion, meme-sharing, and news communities have proportionally more mod discourse (c), while meme and news-sharing communities exhibit the most negative sentiment towards their moderators (d).
• Figure 4: Moderators in communities with lower workloads are perceived more positively and less negatively than moderators in communities with high workloads. Communities with lower moderator workloads (more moderators relative to the amount of content submitted) tend to have more more positive sentiment in their discussion of the moderators, and less negative sentiment. Communities with fewer than five posts and comments per mod per day use $2.5\times$ as much positive sentiment in their mod discourse compared to communities with more than 100 posts and comments per mod per day.
• Figure 5: For most topics, communities where moderators remove more content exhibit more negative sentiment (a). News communities, however, buck this trend, with the fraction of mod discourse with negative sentiment 11 percentage poitns lower in news communities whose mods remove less than $1\%$ of content compared to communities whose mods remove 2% - 3% of content (b).