Enhancing Language Agent Strategic Reasoning through Self-Play in Adversarial Games

TL;DR

The paper tackles the poor strategic reasoning of language agents in dynamic adversarial games by introducing SCO-PAL, a three-stage play-and-learn framework that combines game interaction, step-level reward estimation, and offline strategy refinement via Kahneman-Tversky Optimization. A key contribution is a comprehensive analysis of opponent selection, demonstrating that self-play yields the most balanced and informative training data and the largest performance gains across six GTBench games, including a 54.76% win rate against GPT-4. The approach integrates Behavioral Cloning for fast adaptation with offline preference optimization, and it is shown to generalize to unseen games with meaningful improvements. The work provides practical guidance on opponent selection in adversarial environments and offers a scalable framework for enhancing strategic reasoning in language-based agents.

Abstract

Existing language agents often encounter difficulties in dynamic adversarial games due to poor strategic reasoning. To mitigate this limitation, a promising approach is to allow agents to learn from game interactions automatically, without relying on costly expert-labeled data. Unlike static environments where agents receive fixed feedback or rewards, selecting appropriate opponents in dynamic adversarial games can significantly impact learning performance. However, the discussion of opponents in adversarial environments remains an area under exploration. In this paper, we propose a Step-level poliCy Optimization method through Play-And-Learn, SCO-PAL. Leveraging SCO-PAL, we conduct a detailed analysis of opponent selection by setting opponents at different levels and find that self-play is the most effective way to improve strategic reasoning in such adversarial environments. Utilizing SCO-PAL with self-play, we increase the average win rate against four opponents by approximately 30% compared to baselines and achieve a 54.76% win rate against GPT-4 in six adversarial games.

Figures (9)

• Figure 1: Overview of our play-and-learn method, SCO-PAL. SCO-PAL enables language agents to improve through game interaction via a three-stage process: Stage i Game Interaction assigns an opponent and conducts large-scale gameplay; Stage ii Step-Wise Reward Estimation computes step-level rewards; Stage iii Strategy Refinement updates the policy via behavioral cloning and KTO Optimization.
• Figure 2: Analysis of opponent selection in game interaction. (a) Game Interaction shows the base model's win rates against different opponents, while Evaluation shows the win rates of the model trained with SCO-PAL against four downstream opponents. (b) displays the number of actions collected when interacting with different opponents. The number in MCTS($\cdot$) indicates the max_simulation setting.
• Figure 3: Evaluation of different scaling strategies for balancing advantageous and disadvantageous samples. (a) shows the number of actions obtained under each setting. (b) reports the win rates against four downstream opponents using agents trained on the corresponding data.
• Figure 4: Ablations of SCO-PAL.
• Figure 5: (a) shows the number of actions taken during self-play between strong players of varying levels. In (b), Relative denotes the win rate of the base model against different opponents, while Evaluation indicates the average win rate of the model trained by imitating strong players, evaluated against four downstream opponents. The number in MCTS(·) specifies the max_simulation parameter.