Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Implicit Strategic Optimization: Rethinking Long-Horizon Decision-Making in Adversarial Poker Environments

Boyang Xia, Weiyou Tian, Qingnan Ren, Jiaqi Huang, Jie Xiao, Shuo Lu, Kai Wang, Lynn Ai, Eric Yang, Bill Shi

TL;DR

The paper tackles long-horizon decision-making in adversarial multi-agent settings by identifying that payoffs are shaped by latent, time-evolving strategic externalities. It introduces Implicit Strategic Optimization (ISO), a prediction-aware framework that routes online learning by private context predictions and updates context-specific learners via iso-grpo, a context-conditioned optimistic method. Theoretical results show sublinear contextual regret and convergence to (approximate) coarse correlated equilibria, with dominant terms scaling with context mispredictions and within-context variation. Empirical evaluation in 6-player No-Limit Texas Hold'em and competitive Pokémon demonstrates consistent gains in long-term return over strong LLM and RL baselines, along with graceful degradation under controlled prediction noise. The work provides a principled mechanism to connect forecast quality to long-run performance, offering a scalable path for robust long-horizon decision-making in dynamic, strategic environments.

Abstract

Training large language model (LLM) agents for adversarial games is often driven by episodic objectives such as win rate. In long-horizon settings, however, payoffs are shaped by latent strategic externalities that evolve over time, so myopic optimization and variation-based regret analyses can become vacuous even when the dynamics are predictable. To solve this problem, we introduce Implicit Strategic Optimization (ISO), a prediction-aware framework in which each agent forecasts the current strategic context and uses it to update its policy online. ISO combines a Strategic Reward Model (SRM) that estimates the long-run strategic value of actions with iso-grpo, a context-conditioned optimistic learning rule. We prove sublinear contextual regret and equilibrium convergence guarantees whose dominant terms scale with the number of context mispredictions; when prediction errors are bounded, our bounds recover the static-game rates obtained when strategic externalities are known. Experiments in 6-player No-Limit Texas Hold'em and competitive Pokemon show consistent improvements in long-term return over strong LLM and RL baselines, and graceful degradation under controlled prediction noise.

Implicit Strategic Optimization: Rethinking Long-Horizon Decision-Making in Adversarial Poker Environments

TL;DR

The paper tackles long-horizon decision-making in adversarial multi-agent settings by identifying that payoffs are shaped by latent, time-evolving strategic externalities. It introduces Implicit Strategic Optimization (ISO), a prediction-aware framework that routes online learning by private context predictions and updates context-specific learners via iso-grpo, a context-conditioned optimistic method. Theoretical results show sublinear contextual regret and convergence to (approximate) coarse correlated equilibria, with dominant terms scaling with context mispredictions and within-context variation. Empirical evaluation in 6-player No-Limit Texas Hold'em and competitive Pokémon demonstrates consistent gains in long-term return over strong LLM and RL baselines, along with graceful degradation under controlled prediction noise. The work provides a principled mechanism to connect forecast quality to long-run performance, offering a scalable path for robust long-horizon decision-making in dynamic, strategic environments.

Abstract

Training large language model (LLM) agents for adversarial games is often driven by episodic objectives such as win rate. In long-horizon settings, however, payoffs are shaped by latent strategic externalities that evolve over time, so myopic optimization and variation-based regret analyses can become vacuous even when the dynamics are predictable. To solve this problem, we introduce Implicit Strategic Optimization (ISO), a prediction-aware framework in which each agent forecasts the current strategic context and uses it to update its policy online. ISO combines a Strategic Reward Model (SRM) that estimates the long-run strategic value of actions with iso-grpo, a context-conditioned optimistic learning rule. We prove sublinear contextual regret and equilibrium convergence guarantees whose dominant terms scale with the number of context mispredictions; when prediction errors are bounded, our bounds recover the static-game rates obtained when strategic externalities are known. Experiments in 6-player No-Limit Texas Hold'em and competitive Pokemon show consistent improvements in long-term return over strong LLM and RL baselines, and graceful degradation under controlled prediction noise.
Paper Structure (93 sections, 3 theorems, 26 equations, 7 figures, 19 tables, 1 algorithm)

This paper contains 93 sections, 3 theorems, 26 equations, 7 figures, 19 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. GRPO for LLM Agents
  4. Long-Horizon Adversarial Games and Strategic Learning for LLM Agents
  5. Model
  6. Notation.
  7. Players and actions.
  8. Latent strategic context (what it represents).
  9. Why finite $\mathcal{Z}$.
  10. Linear-in-context loss model (analysis convenience).
  11. Per-action loss vectors under independent play.
  12. Interaction protocol with private context predictions.
  13. Prediction errors.
  14. Prediction-aware regret benchmark (not dynamic regret).
  15. Where a "variation term" can still matter.
  16. ...and 78 more sections

Key Result

Lemma 3.3

For any player $j$ and context $Z$,

Figures (7)

  • Figure 1: Overall ISO framework. (A) ISO training pipeline: expert demonstrations are distilled into an SFT policy, a SRM learns long-horizon strategic value, and ISO-GRPO optimizes the final agent using SRM-guided rewards. (B) ISO-GRPO online mechanism: the agent predicts latent strategic context, routes action selection through the predicted-context policy instance, observes realized context and reward, and updates the corresponding context-specific learner, yielding prediction-aware long-horizon adaptation.
  • Figure 2: Win rate vs. exploitability trade-off.
  • Figure 3: Iterative training improvement in Competitive Pokémon. (a) Cumulative improvement shows win rate against the baseline V0 across training versions. (b) Marginal improvement shows win rate against the immediate predecessor. ISO achieves consistent and monotonic gains.
  • Figure 4: Training Curve of Poker
  • Figure 5: Ablation study of ISO components in 6-player NLHE. Removing SRM, GRPO, or SFT each leads to substantial performance degradation, indicating that all three components are necessary for strong long-horizon strategic performance.
  • ...and 2 more figures

Theorems & Definitions (4)

  • Lemma 3.3: Linearization
  • Proposition 4.1: Contextual regret bound (mispredictions drive regret)
  • Corollary 4.2: Approximate CCE with prediction dependence
  • Example 1: Predictive Counter-Switching under HP Disadvantage