Taming Toxic Talk: Using chatbots to intervene with users posting toxic comments

TL;DR

This study investigates whether rehabilitative conversations generated by GPT-based chatbots can reduce toxic online behavior. In a preregistered field experiment across seven large Reddit communities (N=893 participants; 553 conversations), the authors examine how different persuasive styles shape user engagement and subsequent toxicity. Qualitative analysis of 150 conversations reveals meaningful reflection and varying receptiveness, and Wave 2 shows more reflective and less hostile discussions than Wave 1. However, quantitatively the interventions did not yield significant reductions in toxic behavior or changes in posting activity within a month, highlighting challenges in translating introspective conversations into measurable behavior change and informing future design of AI-assisted moderation and restorative online interventions.

Abstract

Generative AI chatbots have proven surprisingly effective at persuading people to change their beliefs and attitudes in lab settings. However, the practical implications of these findings are not yet clear. In this work, we explore the impact of rehabilitative conversations with generative AI chatbots on users who share toxic content online. Toxic behaviors -- like insults or threats of violence, are widespread in online communities. Strategies to deal with toxic behavior are typically punitive, such as removing content or banning users. Rehabilitative approaches are rarely attempted, in part due to the emotional and psychological cost of engaging with aggressive users. In collaboration with seven large Reddit communities, we conducted a large-scale field experiment (N=893) to invite people who had recently posted toxic content to participate in conversations with AI chatbots. A qualitative analysis of the conversations shows that many participants engaged in good faith and even expressed remorse or a desire to change. However, we did not observe a significant change in toxic behavior in the following month compared to a control group. We discuss possible explanations for our findings, as well as theoretical and practical implications based on our results.

Figures (6)

• Figure 1: Flow of experiment. Users who had a message removed which was higher than the toxicity threshold were invited to chat with the bot. After consent, users were randomized to a condition. Those in the control condition received the handoff message, but did not chat with the bot. Participants in experimental conditions chatted with one of the bots.
• Figure 2: Proportions of reflective and hostile conversations for each condition. The two conditions from Wave 2 were more likely to produce reflective and non-hostile conversations. Error bars refer to standard errors for proportions.
• Figure 3: Regression model results for Wave 1 and Wave 2. Each group on the x-axis represent one mixed-effects regression model, and each point is the point estimate for the beta coefficient for that condition, compared to the control condition. Error bars show the $95\%$ confidence intervals. All values of the Toxicity and Activity models are rescaled by 2 standard deviations of each measure, following gelman_scaling_2008. All models include fixed effects the OpenAI model version and a random intercept for subreddit.
• Figure 4: Regression model results for Wave 1 and Wave 2, aggregating treatment conditions. Each group on the x-axis represent a mixed-effects regression model, and each point is the point estimate for the beta coefficient for that condition, compared to the control condition. Error bars show the $95\%$ confidence intervals. All values of the Toxicity and Activity models are rescaled by 2 standard deviations of each measure, following gelman_scaling_2008. All models include fixed effects the OpenAI model version and a random intercept for subreddit.
• Figure 5: Results showing ridge regression scaled coefficients, under gaussian noise assumptions, for predicting conversation outcomes using behavioral measures during the pre-intervention period. Error bars correspond to the $95\%$ confidence intervals for scaled estimates. Models control for subreddits.