Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Identity, Cooperation and Framing Effects within Groups of Real and Simulated Humans

Suhong Moon, Minwoo Kang, Joseph Suh, Mustafa Safdari, John Canny

TL;DR

This work investigates whether large language models can faithfully simulate human decision making in social dilemmas by capturing social identity and contextual framing. It introduces a framework combining deep persona binding through narrative backstories, Temporal Grounding to anchor simulations in study years, and Consistency Filtering to reduce output drift. Through Dictator and Trust games, the authors show that a DeepBind prompting strategy yields simulations that align closely with human data and enable counterfactual analyses of framing, year, and participant pools. The study discusses biases, data dependence, and resource costs, arguing that LLM based simulations can aid in understanding reproducibility challenges and informing human study design while emphasizing careful ethical considerations and domain limitations.

Abstract

Humans act via a nuanced process that depends both on rational deliberation and also on identity and contextual factors. In this work, we study how large language models (LLMs) can simulate human action in the context of social dilemma games. While prior work has focused on "steering" (weak binding) of chat models to simulate personas, we analyze here how deep binding of base models with extended backstories leads to more faithful replication of identity-based behaviors. Our study has these findings: simulation fidelity vs human studies is improved by conditioning base LMs with rich context of narrative identities and checking consistency using instruction-tuned models. We show that LLMs can also model contextual factors such as time (year that a study was performed), question framing, and participant pool effects. LLMs, therefore, allow us to explore the details that affect human studies but which are often omitted from experiment descriptions, and which hamper accurate replication.

Identity, Cooperation and Framing Effects within Groups of Real and Simulated Humans

TL;DR

This work investigates whether large language models can faithfully simulate human decision making in social dilemmas by capturing social identity and contextual framing. It introduces a framework combining deep persona binding through narrative backstories, Temporal Grounding to anchor simulations in study years, and Consistency Filtering to reduce output drift. Through Dictator and Trust games, the authors show that a DeepBind prompting strategy yields simulations that align closely with human data and enable counterfactual analyses of framing, year, and participant pools. The study discusses biases, data dependence, and resource costs, arguing that LLM based simulations can aid in understanding reproducibility challenges and informing human study design while emphasizing careful ethical considerations and domain limitations.

Abstract

Humans act via a nuanced process that depends both on rational deliberation and also on identity and contextual factors. In this work, we study how large language models (LLMs) can simulate human action in the context of social dilemma games. While prior work has focused on "steering" (weak binding) of chat models to simulate personas, we analyze here how deep binding of base models with extended backstories leads to more faithful replication of identity-based behaviors. Our study has these findings: simulation fidelity vs human studies is improved by conditioning base LMs with rich context of narrative identities and checking consistency using instruction-tuned models. We show that LLMs can also model contextual factors such as time (year that a study was performed), question framing, and participant pool effects. LLMs, therefore, allow us to explore the details that affect human studies but which are often omitted from experiment descriptions, and which hamper accurate replication.
Paper Structure (41 sections, 5 equations, 3 figures, 7 tables)

This paper contains 41 sections, 5 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs for Simulating Human Behavior
  4. Behavioral Games with LLMs
  5. Why Pretrained LLMs ?
  6. Identity Binding in LLM Personas
  7. LLM Personas via Narrative Backstories
  8. Temporal Grounding and Consistency Filtering
  9. Can LLMs Simulate In-/Out-group Biases in Decision Making?
  10. Decision Asymmetry in Dictator Games
  11. Dictator Game.
  12. Experimental Setup and Metrics.
  13. Baselines.
  14. Decision Asymmetry in Trust Games
  15. Trust Game.
  16. ...and 26 more sections

Figures (3)

  • Figure 1: Deep Persona Binding and Consistency Filtering. In this work, pretrained language models are deeply bound with extended backstories, then Temporal Grounding is applied and Consistency Filtering is used to remove implausible or inappropriate responses. The resulting human model more accurately predicts identity driven behaviors and other contextual effects.
  • Figure 2: Conceptual illustration of the differences between pretrained and fine-tuned language models.
  • Figure 3: Per-token perplexity of base and instruction-tuned models on human Reddit data. Instruction-tuned models (orange on right) exhibit substantially higher perplexity across all model families, compared to same-sized pretrained variants (blue on left).