Self-playing Adversarial Language Game Enhances LLM Reasoning

TL;DR

The paper tackles the challenge of improving LLM reasoning by introducing SPAG, a self-play framework around Adversarial Taboo where attacker-defender dialogue is optimized via imitation learning followed by offline reinforcement learning. By modeling the interaction as a zero-sum Markov game and using GPT-4 to bootstrap behavior, SPAG achieves uniform reasoning gains across multiple benchmarks for open-source LLMs and demonstrates progressive improvements over three training epochs. The results suggest that adversarial self-play can substantially enhance general reasoning capabilities while maintaining language fluency, though larger-model validation and safety considerations remain for future work. The work provides a scalable pathway toward deeper, self-improving language agents with potential broad impact across AI applications and ethics.

Abstract

We explore the potential of self-play training for large language models (LLMs) in a two-player adversarial language game called Adversarial Taboo. In this game, an attacker and a defender communicate around a target word only visible to the attacker. The attacker aims to induce the defender to speak the target word unconsciously, while the defender tries to infer the target word from the attacker's utterances. To win the game, both players must have sufficient knowledge about the target word and high-level reasoning ability to infer and express in this information-reserved conversation. Hence, we are curious about whether LLMs' reasoning ability can be further enhanced by Self-Playing this Adversarial language Game (SPAG). With this goal, we select several open-source LLMs and let each act as the attacker and play with a copy of itself as the defender on an extensive range of target words. Through reinforcement learning on the game outcomes, we observe that the LLMs' performances uniformly improve on a broad range of reasoning benchmarks. Furthermore, iteratively adopting this self-play process can continuously promote LLMs' reasoning abilities. The code is available at https://github.com/Linear95/SPAG.

Figures (6)

• Figure 1: Reasoning improvements from Self-Playing of Adversarial language Games (SPAG) on comprehensive reasoning benchmarks. With the SPAG epoch increasing, the LLM reasoning ability continuously improves. Each axis is normalized by the maximum answer-accuracy value.
• Figure 2: Examples of Adversarial Taboo with the same target word "conversation". The left shows an attacker-winning game, in which the defender unconsciously speaks out the target word. The right is a defender-winning episode because the defender makes the correct inference from the dialogue.
• Figure 3: Ablation study of hyper-parameters and data efficiency on imitation learning and first-epoch self-play training. The geometric mean (GM) scores overall reasoning benchmarks are reported. For episode-size ablations, the X-axis is in the logarithmic scale.
• Figure 4: Game results on the testing word list. Left: average win rates of SPAG models playing against GPT-4. Right: average win rate of SPAG attackers against different-epoch checkpoints.
• Figure 5: Game results on the old-version testing word list. Left: average win rates of SPAG models playing against GPT-4. Right: average win rate of SPAG models playing as the attacker against different-epoch checkpoints.