Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

General Social Agents

Benjamin S. Manning, John J. Horton

TL;DR

This work argues that AI agents put in simulations of novel settings offer an alternative for applying theory, requiring minimal or no modifications, and presents an approach for building general AI agents that use theory-grounded natural language instructions, existing empirical data, and knowledge acquired by the underlying AI during training.

Abstract

Useful social science theories predict behavior across settings. However, applying a theory to make predictions in new settings is challenging: rarely can it be done without ad hoc modifications to account for setting-specific factors. We argue that AI agents put in simulations of those novel settings offer an alternative for applying theory, requiring minimal or no modifications. We present an approach for building such "general" agents that use theory-grounded natural language instructions, existing empirical data, and knowledge acquired by the underlying AI during training. To demonstrate the approach in settings where no data from that data-generating process exists--as is often the case in applied prediction problems--we design a heterogeneous population of 883,320 novel games. AI agents are constructed using human data from a small set of conceptually related but structurally distinct "seed" games. In preregistered experiments, on average, agents predict initial human play in a random sample of 1,500 games from the population better than (i) a cognitive hierarchy model, (ii) game-theoretic equilibria, and (iii) out-of-the-box agents. For a small set of separate novel games, these simulations predict responses from a new sample of human subjects better even than the most plausibly relevant published human data.

General Social Agents

TL;DR

This work argues that AI agents put in simulations of novel settings offer an alternative for applying theory, requiring minimal or no modifications, and presents an approach for building general AI agents that use theory-grounded natural language instructions, existing empirical data, and knowledge acquired by the underlying AI during training.

Abstract

Useful social science theories predict behavior across settings. However, applying a theory to make predictions in new settings is challenging: rarely can it be done without ad hoc modifications to account for setting-specific factors. We argue that AI agents put in simulations of those novel settings offer an alternative for applying theory, requiring minimal or no modifications. We present an approach for building such "general" agents that use theory-grounded natural language instructions, existing empirical data, and knowledge acquired by the underlying AI during training. To demonstrate the approach in settings where no data from that data-generating process exists--as is often the case in applied prediction problems--we design a heterogeneous population of 883,320 novel games. AI agents are constructed using human data from a small set of conceptually related but structurally distinct "seed" games. In preregistered experiments, on average, agents predict initial human play in a random sample of 1,500 games from the population better than (i) a cognitive hierarchy model, (ii) game-theoretic equilibria, and (iii) out-of-the-box agents. For a small set of separate novel games, these simulations predict responses from a new sample of human subjects better even than the most plausibly relevant published human data.

Paper Structure

This paper contains 43 sections, 1 theorem, 9 equations, 20 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Building prompts that generalize
  3. A motivating example
  4. Identifying generalizable behavioral relationships
  5. Theory as guide towards generalizability
  6. A brief summary of the approach
  7. Methods for optimizing prompts in-sample
  8. Selection Method.
  9. Construction Method.
  10. Measuring Performance.
  11. Predicting behavior in novel strategic games
  12. 1120Arad2012's 11-20 money request game
  13. Optimizing AI agents in-sample
  14. Validation using game variants
  15. Optimizing among atheoretical prompts
  16. ...and 28 more sections

Key Result

Proposition 1

Suppose Assumptions ass:rand-settings--ass:response-support hold. Then where $\sigma^{2} = \operatorname{Var}_{x\sim\pi}\!\bigl[\Lambda(x)\bigr] \;+\; \mathbb{E}_{x\sim\pi}\!\Bigl[\tfrac{1}{m_x}\,V_x\Bigr]$ and $V_x = \operatorname{Var}_{y \sim P(\cdot \mid x)} \bigl[\log\hat{P}_{\theta'}(y \mid x) -\log\hat{P}_{\theta"}( y\mid x)\bigr].$Proof. (Unbiasedness). For any

Figures (20)

  • Figure 1: Response distributions for the basic version of the 11-20 game
  • Figure 2: Response distributions for the cycle and costless versions of the 11-20 game
  • Figure 3: Response distributions for the 11-20 games with atheoretical AI agents
  • Figure 4: Analysis of novel 1-10 games: response distributions and KL divergences
  • Figure 5: Predictive distributions from the cognitive hierarchy model for all 1,500 games in $S$
  • ...and 15 more figures

Theorems & Definitions (1)

  • Proposition 1: Unbiasedness and asymptotic normality