Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction

Nils Schwager, Simon Münker, Alistair Plum, Achim Rettinger

TL;DR

This study introduces Conditioned Comment Prediction (CCP), a task in which a model predicts how a user would comment on a given stimulus by comparing generated outputs with authentic digital traces, and offers operational guidelines prioritizing authentic behavioral traces over descriptive personas for high-fidelity simulation.

Abstract

The transition of Large Language Models (LLMs) from exploratory tools to active "silicon subjects" in social science lacks extensive validation of operational validity. This study introduces Conditioned Comment Prediction (CCP), a task in which a model predicts how a user would comment on a given stimulus by comparing generated outputs with authentic digital traces. This framework enables a rigorous evaluation of current LLM capabilities with respect to the simulation of social media user behavior. We evaluated open-weight 8B models (Llama3.1, Qwen3, Ministral) in English, German, and Luxembourgish language scenarios. By systematically comparing prompting strategies (explicit vs. implicit) and the impact of Supervised Fine-Tuning (SFT), we identify a critical form vs. content decoupling in low-resource settings: while SFT aligns the surface structure of the text output (length and syntax), it degrades semantic grounding. Furthermore, we demonstrate that explicit conditioning (generated biographies) becomes redundant under fine-tuning, as models successfully perform latent inference directly from behavioral histories. Our findings challenge current "naive prompting" paradigms and offer operational guidelines prioritizing authentic behavioral traces over descriptive personas for high-fidelity simulation.

Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction

TL;DR

This study introduces Conditioned Comment Prediction (CCP), a task in which a model predicts how a user would comment on a given stimulus by comparing generated outputs with authentic digital traces, and offers operational guidelines prioritizing authentic behavioral traces over descriptive personas for high-fidelity simulation.

Abstract

The transition of Large Language Models (LLMs) from exploratory tools to active "silicon subjects" in social science lacks extensive validation of operational validity. This study introduces Conditioned Comment Prediction (CCP), a task in which a model predicts how a user would comment on a given stimulus by comparing generated outputs with authentic digital traces. This framework enables a rigorous evaluation of current LLM capabilities with respect to the simulation of social media user behavior. We evaluated open-weight 8B models (Llama3.1, Qwen3, Ministral) in English, German, and Luxembourgish language scenarios. By systematically comparing prompting strategies (explicit vs. implicit) and the impact of Supervised Fine-Tuning (SFT), we identify a critical form vs. content decoupling in low-resource settings: while SFT aligns the surface structure of the text output (length and syntax), it degrades semantic grounding. Furthermore, we demonstrate that explicit conditioning (generated biographies) becomes redundant under fine-tuning, as models successfully perform latent inference directly from behavioral histories. Our findings challenge current "naive prompting" paradigms and offer operational guidelines prioritizing authentic behavioral traces over descriptive personas for high-fidelity simulation.
Paper Structure (73 sections, 1 figure, 7 tables)

This paper contains 73 sections, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Research Questions
  3. Contributions
  4. Multilingual Benchmarking of Comment Prediction
  5. Evaluating Conditioning Strategies
  6. Operational Guidelines
  7. Background
  8. LLMs as Agents in Social Simulations
  9. Prompting Social Media Users
  10. Explicit
  11. Implicit
  12. Methods for Conditioned Comment Prediction
  13. Task Definition
  14. Conditioning Strategies
  15. User History (Implicit)
  16. ...and 58 more sections

Figures (1)

  • Figure 1: Impact of history length on predictive performance. Results illustrate the dependence between the volume of provided behavioral history (number of previous comments) and prediction quality. Shaded regions represent the standard deviation across 5 independent generation runs, while the underlying gray bars indicate the sample size distribution per length bucket. Analysis is based on Llama-3.1-8B in English using the History-Only conditioning strategy. The fine-tuned model was trained on a dataset of 3,800 users.