Does Positive Reinforcement Work?: A Quasi-Experimental Study of the Effects of Positive Feedback on Reddit

TL;DR

The paper investigates whether positive reinforcement through Reddit's signals affects user behavior, using a large-scale causal-inference approach. It applies a potential outcomes framework with stratified matching and difference-in-differences to 11 million posts across four months, comparing posts that receive gold or high upvotes against similar, untreated posts. The results show that recipients of positive feedback generate higher-quality, more positively received content, with larger effects for stronger reinforcement and for newcomers, and that these effects persist differently over time. The findings offer actionable guidance for platforms and moderators to blend positive reinforcement with traditional punitive moderation, aiming to sustain user motivation and norm adherence in online communities.

Abstract

Social media platform design often incorporates explicit signals of positive feedback. Some moderators provide positive feedback with the goal of positive reinforcement, but are often unsure of their ability to actually influence user behavior. Despite its widespread use and theory touting positive feedback as crucial for user motivation, its effect on recipients is relatively unknown. This paper examines how positive feedback impacts Reddit users and evaluates its differential effects to understand who benefits most from receiving positive feedback. Through a causal inference study of 11M posts across 4 months, we find that users who received positive feedback made more frequent (2% per day) and higher quality (57% higher score; 2% fewer removals per day) posts compared to a set of matched control users. Our findings highlight the need for platforms, communities, and moderators to expand their perspective on moderation and complement punitive approaches with positive reinforcement strategies to foster desirable behavior online.

Figures (5)

• Figure 1: This is an example post from r/aww that received gold, represented by a small gold icon above the title. The post's score is reported under its title. This post's score was in the top 25th percentile for r/aww in the month it was posted.
• Figure 2: Pipeline from data collection to forming matched treated and control samples used in our causal inference methodology.
• Figure 3: Standardized mean differences (SMD) for each covariate averaged across all strata. We report both SMDs for matched data and unmatched data for comparison. The figure shows that matching improved the SMD for most covariates and that the SMD for all matched data is below the threshold 0.3, indicating high-quality matches.
• Figure 4: Visualization of each outcome over the baseline and observation windows (excluding treatment/placebo posts themselves). These plots are used to validate the parallel trends assumption needed for difference-in-differences analysis.
• Figure 5: Average treatment effects (ATE) of each treatment on the five outcomes. The rightmost $y$-axes correspond to the ATE of the score outcome which has a much larger range. The "x" markers indicate the days at which each outcome measure reaches saturation (i.e., decreases or stays the same for two consecutive days). The figure demonstrates the lasting effect of the score treatment on each outcome compared to the gold treatment, where the ATEs mostly return to zero within two weeks.