Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Next Reply Prediction X Dataset: Linguistic Discrepancies in Naively Generated Content

Simon Münker, Nils Schwager, Kai Kugler, Michael Heseltine, Achim Rettinger

TL;DR

This paper investigates whether Large Language Models can reliably mimic authentic social-media behavior, introducing a history-conditioned reply-prediction dataset across English and German to quantify linguistic discrepancies between human and synthetic content. It develops a multi-dimensional evaluation framework—covering quantitative features, morphosyntactic patterns, semantic classifications, cluster-based similarity, and feature-vector distances—and compares base and fine-tuned model generations. Results show that fine-tuned outputs align more closely with human content than prompt-based ones, yet persistent detectable signatures remain, challenging ecological validity. The study advocates mandatory validation protocols, domain-specific fine-tuning, and multi-dimensional evaluation to improve the realism and reliability of LLM-based social simulations in computational social science.

Abstract

The increasing use of Large Language Models (LLMs) as proxies for human participants in social science research presents a promising, yet methodologically risky, paradigm shift. While LLMs offer scalability and cost-efficiency, their "naive" application, where they are prompted to generate content without explicit behavioral constraints, introduces significant linguistic discrepancies that challenge the validity of research findings. This paper addresses these limitations by introducing a novel, history-conditioned reply prediction task on authentic X (formerly Twitter) data, to create a dataset designed to evaluate the linguistic output of LLMs against human-generated content. We analyze these discrepancies using stylistic and content-based metrics, providing a quantitative framework for researchers to assess the quality and authenticity of synthetic data. Our findings highlight the need for more sophisticated prompting techniques and specialized datasets to ensure that LLM-generated content accurately reflects the complex linguistic patterns of human communication, thereby improving the validity of computational social science studies.

Next Reply Prediction X Dataset: Linguistic Discrepancies in Naively Generated Content

TL;DR

This paper investigates whether Large Language Models can reliably mimic authentic social-media behavior, introducing a history-conditioned reply-prediction dataset across English and German to quantify linguistic discrepancies between human and synthetic content. It develops a multi-dimensional evaluation framework—covering quantitative features, morphosyntactic patterns, semantic classifications, cluster-based similarity, and feature-vector distances—and compares base and fine-tuned model generations. Results show that fine-tuned outputs align more closely with human content than prompt-based ones, yet persistent detectable signatures remain, challenging ecological validity. The study advocates mandatory validation protocols, domain-specific fine-tuning, and multi-dimensional evaluation to improve the realism and reliability of LLM-based social simulations in computational social science.

Abstract

The increasing use of Large Language Models (LLMs) as proxies for human participants in social science research presents a promising, yet methodologically risky, paradigm shift. While LLMs offer scalability and cost-efficiency, their "naive" application, where they are prompted to generate content without explicit behavioral constraints, introduces significant linguistic discrepancies that challenge the validity of research findings. This paper addresses these limitations by introducing a novel, history-conditioned reply prediction task on authentic X (formerly Twitter) data, to create a dataset designed to evaluate the linguistic output of LLMs against human-generated content. We analyze these discrepancies using stylistic and content-based metrics, providing a quantitative framework for researchers to assess the quality and authenticity of synthetic data. Our findings highlight the need for more sophisticated prompting techniques and specialized datasets to ensure that LLM-generated content accurately reflects the complex linguistic patterns of human communication, thereby improving the validity of computational social science studies.
Paper Structure (55 sections, 2 figures, 2 tables)

This paper contains 55 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Research Questions and Hypotheses
  3. Our Contributions
  4. Background
  5. LLMs as Human Simulacra
  6. Synthetic Content Detection in Social Media
  7. Methods
  8. Data: Authentic vs. Synthetic
  9. Collection/Preprocessing
  10. Transformation
  11. Fine-Tuning
  12. Generation
  13. Published Dataset
  14. Evaluation: Levels of Alignment
  15. Quantitative Features
  16. ...and 40 more sections

Figures (2)

  • Figure 1: Locality, spread and skewness (x-axes) of each POS category (y-axes) for the English corpus split into subsets.
  • Figure 2: Locality, spread and skewness (y-axes) of the TweetEval topic and emotion classifier (x-axes) for the English corpus split into subsets.