More at Stake: How Payoff and Language Shape LLM Agent Strategies in Cooperation Dilemmas

TL;DR

This work investigates how payoff incentives and linguistic framing shape LLMs' strategic behavior in repeated social dilemmas, using a payoff-scaled Prisoner’s Dilemma implemented via the FAIRGAME framework. It couples multilingual prompting with an intention-recognition pipeline (trained on canonical strategies ALLC, ALLD, TFT, WSLS) to infer underlying strategies from 10-round trajectories across three models and five languages. The study finds systematic payoff sensitivity and robust cross-language biases, with higher stakes promoting cooperative/conditional strategies and language-specific priming of strategy types; model architecture modulates these effects, revealing diverse strategic priors. These results provide a governance-relevant diagnostic toolkit for auditing LLMs as strategic agents, underscoring the need to evaluate safety and coordination across incentive structures and linguistic contexts in multi-agent deployments.

Abstract

As LLMs increasingly act as autonomous agents in interactive and multi-agent settings, understanding their strategic behavior is critical for safety, coordination, and AI-driven social and economic systems. We investigate how payoff magnitude and linguistic context shape LLM strategies in repeated social dilemmas, using a payoff-scaled Prisoner's Dilemma to isolate sensitivity to incentive strength. Across models and languages, we observe consistent behavioral patterns, including incentive-sensitive conditional strategies and cross-linguistic divergence. To interpret these dynamics, we train supervised classifiers on canonical repeated-game strategies and apply them to LLM decisions, revealing systematic, model- and language-dependent behavioral intentions, with linguistic framing sometimes matching or exceeding architectural effects. Our results provide a unified framework for auditing LLMs as strategic agents and highlight cooperation biases with direct implications for AI governance and multi-agent system design.

Figures (15)

• Figure 1: Supervised Learning Pipeline for Understanding LLM behavior. Starting from action sequences associated with canonical strategies (ALLC, ALLD, TFT, WSLS) under varying noise conditions, we train supervised learning models to infer and classify underlying behavioral intentions. We then apply the best-performing model to the LLM repeated gameplay data generated by FAIRGAME. High-confidence predictions ($>$0.9) are used to identify which strategies the LLM adopts, whereas low-confidence cases are reserved for subsequent analysis to investigate the possibility of emerging behaviors by the LLM.
• Figure 2: Aggregated final penalties across repeated Prisoner’s Dilemma games, presented for each LLM under different payoff scales. Results are reported for five languages, and evaluated across the payoff-scaling parameters $\lambda \in \{0.1, 1.0, 10.0\}$, which correspond to attenuated (top row), baseline (middle row), and amplified penalty scales (bottom row), respectively.
• Figure 3: Average trajectory of strategy choices across repeated rounds in all Prisoner's Dilemma experiments, shown for each LLM under different payoff magnitudes. A value of $1$ indicates selection of Option A (defection), while $-1$ corresponds to Option B (cooperation). The experiments consider $\lambda \in \{0.1, 1.0, 10.0\}$, representing attenuated, baseline, and amplified penalty scales, respectively. The blue line denotes the standard payoff matrix ($\lambda = 1.0$), the red line reflects the payoff matrix scaled by $10$ ($\lambda = 10.0$), and the green line represents the payoff matrix scaled by $0.1$ ($\lambda = 0.1$).
• Figure 4: LSTM intent recognizer performance. F1 ranges from 0.78 (5-strategy) to 0.984 (4-strategy) on 5% noise data; all subsequent analyses use the 4-strategy classifier (F1$=$0.984).
• Figure 5: Trends in inferred strategy distributions as a function of the payoff scaling parameter $\lambda$. Increasing payoff magnitude systematically shifts LLM behavior from unconditional defection (ALLD) toward conditional and cooperative strategies (WSLS, ALLC), indicating sensitivity to incentive scale.