Machines in the Crowd? Measuring the Footprint of Machine-Generated Text on Reddit

TL;DR

The paper addresses how machine-generated text diffuses within Reddit by analyzing 51 subreddits from 2022–2024 using a zero-shot detector, Fast-DetectGPT, to label MGT with high precision. It combines content analysis of style and social dimensions with engagement comparisons (using Mann–Whitney tests and Cliff’s delta) to reveal that MGT, while unevenly distributed and concentrated among a small user base, is non-negligible and can mimic or outperform human-authored text in engagement. The study finds MGT more prevalent in information-seeking and identity communities, with temporal peaks tied to major GenAI tool releases, and discovers that MGT often signals warmth and status. These findings have implications for platform governance, authenticity, and the co-evolution of human and AI voices in online discourse, underscoring the need for ongoing monitoring and robust detection strategies.

Abstract

Generative Artificial Intelligence is reshaping online communication by enabling large-scale production of Machine-Generated Text (MGT) at low cost. While its presence is rapidly growing across the Web, little is known about how MGT integrates into social media environments. In this paper, we present the first large-scale characterization of MGT on Reddit. Using a state-of-the-art statistical method for detection of MGT, we analyze over two years of activity (2022-2024) across 51 subreddits representative of Reddit's main community types such as information seeking, social support, and discussion. We study the concentration of MGT across communities and over time, and compared MGT to human-authored text in terms of social signals it expresses and engagement it receives. Our very conservative estimate of MGT prevalence indicates that synthetic text is marginally present on Reddit, but it can reach peaks of up to 9% in some communities in some months. MGT is unevenly distributed across communities, more prevalent in subreddits focused on technical knowledge and social support, and often concentrated in the activity of a small fraction of users. MGT also conveys distinct social signals of warmth and status giving typical of language of AI assistants. Despite these stylistic differences, MGT achieves engagement levels comparable than human-authored content and in a few cases even higher, suggesting that AI-generated text is becoming an organic component of online social discourse. This work offers the first perspective on the MGT footprint on Reddit, paving the way for new investigations involving platform governance, detection strategies, and community dynamics.

Figures (8)

• Figure 1: MGT comment usage across subreddits. Rows are sorted by the sum of monthly MGT adoption. Each cell shows the share of MGT posts for a subreddit-month pair. Darker shades indicate higher usage. Empty cells mean no comments matching our filtering criteria.
• Figure 2: Average number of users adopting MGT (in blue, left y-axis) and the corresponding percentage of their comments detected to be MGT (in red, right y-axis) across all categories over time. Shaded areas denote minimum and maximum observed values. Users with a single post have been filtered out to mitigate noise due to one-off activities.
• Figure 3: Comparison of social dimension expressions between MGT (in red) and HGT (in blue) comments across different subreddit categories. Each subplot shows the distribution of social dimension scores $\phi_d(x)$ (excluding instances with $\phi_d(x)=0$) for all combinations of subreddit and social dimension. Each subplot is labeled with the corresponding effect size, where $>$ indicates that MGT scores higher than HGT, and $<$ indicates the opposite. Marker $^{***}$ denotes that the difference in social dimension expression scores between MGT and HGT is statistically significant according to the Mann–Whitney U test.
• Figure A1: Monthly volume of comments for each subreddit category.
• Figure A2: Quarterly volume of submissions for each subreddit category.