Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Collective Turing Test: Large Language Models Can Generate Realistic Multi-User Discussions

Azza Bouleimen, Giordano De Marzo, Taehee Kim, Nicol`o Pagan, Hannah Metzler, Silvia Giordano, David Garcia

TL;DR

It is demonstrated that LLMs can generate social media conversations sufficiently realistic to deceive humans when reading them, highlighting both a promising potential for social simulation and a warning message about the potential misuse of LLMs to generate new inauthentic social media content.

Abstract

Large Language Models (LLMs) offer new avenues to simulate online communities and social media. Potential applications range from testing the design of content recommendation algorithms to estimating the effects of content policies and interventions. However, the validity of using LLMs to simulate conversations between various users remains largely untested. We evaluated whether LLMs can convincingly mimic human group conversations on social media. We collected authentic human conversations from Reddit and generated artificial conversations on the same topic with two LLMs: Llama 3 70B and GPT-4o. When presented side-by-side to study participants, LLM-generated conversations were mistaken for human-created content 39\% of the time. In particular, when evaluating conversations generated by Llama 3, participants correctly identified them as AI-generated only 56\% of the time, barely better than random chance. Our study demonstrates that LLMs can generate social media conversations sufficiently realistic to deceive humans when reading them, highlighting both a promising potential for social simulation and a warning message about the potential misuse of LLMs to generate new inauthentic social media content.

The Collective Turing Test: Large Language Models Can Generate Realistic Multi-User Discussions

TL;DR

It is demonstrated that LLMs can generate social media conversations sufficiently realistic to deceive humans when reading them, highlighting both a promising potential for social simulation and a warning message about the potential misuse of LLMs to generate new inauthentic social media content.

Abstract

Large Language Models (LLMs) offer new avenues to simulate online communities and social media. Potential applications range from testing the design of content recommendation algorithms to estimating the effects of content policies and interventions. However, the validity of using LLMs to simulate conversations between various users remains largely untested. We evaluated whether LLMs can convincingly mimic human group conversations on social media. We collected authentic human conversations from Reddit and generated artificial conversations on the same topic with two LLMs: Llama 3 70B and GPT-4o. When presented side-by-side to study participants, LLM-generated conversations were mistaken for human-created content 39\% of the time. In particular, when evaluating conversations generated by Llama 3, participants correctly identified them as AI-generated only 56\% of the time, barely better than random chance. Our study demonstrates that LLMs can generate social media conversations sufficiently realistic to deceive humans when reading them, highlighting both a promising potential for social simulation and a warning message about the potential misuse of LLMs to generate new inauthentic social media content.

Paper Structure

This paper contains 15 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Results
  3. Discussion
  4. Conclusion
  5. Methods
  6. Acknowledgments
  7. Funding statement
  8. Author contributions statement
  9. Competing interests
  10. Data availability statement

Figures (3)

  • Figure 1: Left: Predicted probabilities of correctly identifying the human conversation by model. Right: Estimates of fixed effects in the logistic regression model with length as a numerical variable. 95% confidence intervals are shown and bars in orange are significant at the 95% level.
  • Figure 2: Left: Predicted probabilities of correctly identifying the human conversation by model = GPT-4o and length as a categorical variable. Center: Predicted probabilities of correctly identifying the human conversation by model = Llama 3 70B and length as a categorical variable. Right: Plot of the fixed effect of the logistic regression when length is a categorical variable. Reference is Length = 1 comment. Coefficients with the orange CI are statistically significant.
  • Figure 3: Example of annotations shown to participants.