Introducing AI to an Online Petition Platform Changed Outputs but not Outcomes

TL;DR

The results suggest that while AI writing tools can profoundly reshape online content, their practical utility for improving desired outcomes may be less beneficial than anticipated, and introduce unintended consequences like content homogenization.

Abstract

The rapid integration of AI writing tools into online platforms raises critical questions about their impact on content production and outcomes. We leverage a unique natural experiment on Change$.$org, a leading social advocacy platform, to causally investigate the effects of an in-platform ''write with AI'' tool. To understand the impact of the AI integration, we collected 1.5 million petitions and employed a difference-in-differences analysis. Our findings reveal that in-platform AI access significantly altered the lexical features of petitions and increased petition homogeneity, but did not improve petition outcomes. We confirmed the results in a separate analysis of repeat petition writers who wrote petitions before and after introduction of the AI tool. The results suggest that while AI writing tools can profoundly reshape online content, their practical utility for improving desired outcomes may be less beneficial than anticipated, and introduce unintended consequences like content homogenization.

Figures (27)

• Figure 1: Platform trends in lexical features (a, b) changed significantly with AI access while trends in outcomes (c, d) did not improve, or worsened. Difference-in-differences (DiD) analysis of platform trends in petitions' lexical features (a, b) and outcomes (c,d) before and after Change.org AI tool release. Change in outcomes reflect change in percentage points. (a) Dynamic difference-in-differences with estimated effect of AI access on lexical features by week, relative to the baseline week (Week -1). (b) Aggregated estimated effect of AI access on lexical features, relative to pre-AI period. (c) Dynamic difference-in-differences with estimated effect of AI access on outcomes by week. (d) Aggregated estimated effect of AI access on outcomes, relative to pre-AI period. The A/B testing period, shown in gray between the dashed lines, is excluded from analysis. 95% confidence intervals are shown. Pre-AI period was 64 weeks (N = 235,947), A/B test period was 27 weeks (N = 82,781), and the Post-AI period was 10 weeks (N = 42,100).
• Figure 2: Dynamic difference-in-difference estimates for homogeneity in countries with access to AI shows that with access to the AI tool, platform homogeneity increases. Estimated average treatment effect (ATT) of mean pairwise similarity is shown with 95% CIs. Estimates are compared to the baseline the week prior to A/B testing in treated countries (Week -1). Pre-AI period was 64 weeks (N = 235,947), A/B test period was 27 weeks (N = 82,781), and the Post-AI period was 10 weeks (N = 42,100).
• Figure 3: Returning petition writers (N=4,553 users) from treated countries, on average, write longer petitions with worse outcomes when their second petition is written with access to AI. (a) Petition length is higher in petition writers' second petitions, in all conditions. The greatest increase in word count occurs for writers in the Pre/Post split cohort. Average word count is shown in black with 95% CI's. (b) Petition outcomes are worse in writers' second petitions, in all conditions. The greatest drop in outcomes for users' second petitions occurs when the second petition is written with access to AI. 95% CI's (boostrapped, 1,000 iterations) are shown. Returning petition writers who return within 6 months are excluded, to enforce the same minimum time gap between petitions across cohorts.
• Figure S1: Change.org home screen that advertises the site's AI feature as capable of creating compelling petitions in minutes. Screenshot taken on February 14, 2025.
• Figure S2: AI detection on English petitions (any length) from treated and control countries (N = 482,846). The share of AI-generated petitions increases sharply after the introduction of the AI-generated draft feature in countries with access to the tool. Countries with access to the tool are the United States (US) Canada (CA), and Great Britain (GB), the country without access to the tool is Australia (AU). As of December, 2023, most (60%) of petitions written with access to the AI-generated draft feature on Change.org were generated with AI. Plot shows AI classification results from January 1, 2022 to December 15, 2023. Our ensemble classifier achieved 99% accuracy, 98% precision, and 99% recall on a test set of 1,200 annotated human and AI petitions.