Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GameTalk: Training LLMs for Strategic Conversation

Victor Conchello Vendrell, Max Ruiz Luyten, Mihaela van der Schaar

TL;DR

GameTalk introduces a framework to train LLMs for strategic, multi-turn dialogue by optimizing a global objective across entire conversations. It adapts three fine-tuning methods—GRPO, DPO, and STaR—to interactive settings and couples them with three behavioral signals, ISE, SRP, and LO, for reward shaping that drives long-horizon decision making. Across Rock–Paper–Scissors, Bertrand Competition, and Size–Price Bargaining, the approach yields consistent improvements over untrained baselines, with DPO delivering the strongest gains in more complex tasks. The work demonstrates that conversational fine-tuning can enable LLMs to reason, negotiate, and influence opponent behavior in interactive environments, offering a principled path toward robust strategic dialogue while highlighting challenges in opponent modeling and generalization to humans.

Abstract

Strategic decision-making in multi-agent settings is a key challenge for large language models (LLMs), particularly when coordination and negotiation must unfold over extended conversations. While recent work has explored the use of LLMs in isolated decision tasks, little attention has been given to optimizing long-term objectives through dialogue. We introduce \textbf{GameTalk}, a framework for training LLMs to make strategic decisions via multi-turn interactions. Unlike prior work that focuses on single-turn objectives or static action prediction, we train LLMs to optimize a global objective across full conversations. We achieve this by adapting fine-tuning methods like GRPO, DPO, and STaR to incorporate reward signals that depend on the entire interaction. We evaluate this approach on a suite of increasingly complex games, designed to stress different aspects of reasoning, coordination, and opponent modeling. Our results show that GameTalk significantly outperforms untrained models, especially under reward shaping, with DPO consistently yielding the strongest gains. These findings position conversational fine-tuning as a promising path for LLMs to reason, negotiate, and act in interactive environments.

GameTalk: Training LLMs for Strategic Conversation

TL;DR

GameTalk introduces a framework to train LLMs for strategic, multi-turn dialogue by optimizing a global objective across entire conversations. It adapts three fine-tuning methods—GRPO, DPO, and STaR—to interactive settings and couples them with three behavioral signals, ISE, SRP, and LO, for reward shaping that drives long-horizon decision making. Across Rock–Paper–Scissors, Bertrand Competition, and Size–Price Bargaining, the approach yields consistent improvements over untrained baselines, with DPO delivering the strongest gains in more complex tasks. The work demonstrates that conversational fine-tuning can enable LLMs to reason, negotiate, and influence opponent behavior in interactive environments, offering a principled path toward robust strategic dialogue while highlighting challenges in opponent modeling and generalization to humans.

Abstract

Strategic decision-making in multi-agent settings is a key challenge for large language models (LLMs), particularly when coordination and negotiation must unfold over extended conversations. While recent work has explored the use of LLMs in isolated decision tasks, little attention has been given to optimizing long-term objectives through dialogue. We introduce \textbf{GameTalk}, a framework for training LLMs to make strategic decisions via multi-turn interactions. Unlike prior work that focuses on single-turn objectives or static action prediction, we train LLMs to optimize a global objective across full conversations. We achieve this by adapting fine-tuning methods like GRPO, DPO, and STaR to incorporate reward signals that depend on the entire interaction. We evaluate this approach on a suite of increasingly complex games, designed to stress different aspects of reasoning, coordination, and opponent modeling. Our results show that GameTalk significantly outperforms untrained models, especially under reward shaping, with DPO consistently yielding the strongest gains. These findings position conversational fine-tuning as a promising path for LLMs to reason, negotiate, and act in interactive environments.
Paper Structure (59 sections, 2 theorems, 7 equations, 8 figures, 2 tables)

This paper contains 59 sections, 2 theorems, 7 equations, 8 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. AIs in game settings
  4. Beyond Rational Agents in Game Theory
  5. METHOD
  6. GameTalk
  7. Behavioral Signals for Strategic Analysis
  8. Adaptation of RL alogrithms
  9. Group Relative Policy Optimization shaoDeepSeekMathPushingLimits2024
  10. Direct Preference Optimization rafailovDirectPreferenceOptimization2024
  11. Self-Taught Reasoner zelikmanSTaRBootstrappingReasoning2022
  12. EXPERIMENTS
  13. Reward Shaping Analysis
  14. Setup.
  15. Training Configurations.
  16. ...and 44 more sections

Key Result

Theorem 1

Let $E_{true}[u_i]$ be the agent's true expected utility, determined by its policy and its opponent's true policy. If the behavioral metrics $ISE$, $SRP$, and $LO$ are computed using the exact opponent policy $\pi_{true}$ and agent belief $\pi_{belief}$, then the utility is bounded as follows: where $C$ is a constant representing the maximum range of the utility function $u_i$.

Figures (8)

  • Figure 1: Example of a game of rock-paper-scissors. All texts have been shortened. The game starts with a setting prompt. The two agents first engage in a conversation, using the communication action. When the conversation is over, they do the game action. That concludes the game. Before each of these actions, they use the private CoT, to decide their actions, this is not shown to the opposite LLM. At the end of the game, one LLM is trained using the reward obtained from this episode.
  • Figure 2: Illustration of how $\hat{\pi}_{true}$ and $\hat{\pi}_{belief}$ are obtained, in order to use them in ISE, SRP and LO. Note that they are approximations, since they have to be computed without the Private CoT, which would influence the final action.
  • Figure 3: Illustration of the generation process used in GRPO and DPO. The first three interactions correspond to the root conversation, which is then duplicated into $k=3$ parallel copies. Each copy is completed independently, and the reward obtained from each is used to train the policy on the first response after the copying step.
  • Figure 4: Analysis of reward shaping in the constrained Rock-Paper-Scissors game. Left: Spider plot with general metrics (rescaled for visualization), comparing an Untrained model, a Base agent trained only on game reward, and agents trained with ISE and LO auxiliary rewards. Middle: Bar chart comparing win/draw/lose rates of all models. Right: Spider plot comparing Base and LO-reward against an agent trained with LO-reward and Naturalness reward. Bottom: Table with the raw values of the plots, bold for the best value and underline for the second best.
  • Figure 5: Comparative analysis of training algorithms across the three game environments when applied with the GameTalk framework. Performance metrics are shown for (a) Rock-Paper-Scissors, (b) Bertrand Competition, and (c) Size-Price Bargaining. For each game, tables report exact metric values (best in bold, and second best underlined), and spider plots visualize the key trade-offs between reward ($R$), game-specific outcomes (Win %, $NE$, $BP$), and our behavioral signals. The bar chart in (a) details win/draw/lose rates for Rock-Paper-Scissors.
  • ...and 3 more figures

Theorems & Definitions (3)

  • Theorem 1: Utility Bounds from Behavioral Signals
  • Theorem 2: Utility Bounds from Behavioral Signals
  • proof