GIFT: Games as Informal Training for Generalizable LLMs

TL;DR

The paper tackles the lack of informal learning in LLMs by proposing games as scalable informal learning environments and introducing a nested training framework that replaces the traditional OR-style multi-task objective with an explicit AND objective, formalized as $\max_{\theta} \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau_1,\dots,\tau_K)]$. It implements GRPO-based reinforcement learning to integrate formal math tasks with game environments (Matrix Games, TicTacToe, Who’s the Spy), and demonstrates that nested training yields superior generalization and training stability compared to naive mixed training, with notable gains on both 1.5B and 7B models (e.g., average general ability rising from 38.34% to 42.43% for 1.5B and from 42.00% to 55.84% for 7B). Case studies show that informal learning fosters explicit, verifiable reasoning in math and deeper semantic integration in language generation. The results indicate that carefully designed interactive training objectives can elevate generalizable capabilities in smaller LLMs, offering a scalable path toward more robust AI systems; the work also provides practical guidelines for combining formal and informal learning in future models.

Abstract

While Large Language Models (LLMs) have achieved remarkable success in formal learning tasks such as mathematics and code generation, they still struggle with the "practical wisdom" and generalizable intelligence, such as strategic creativity and social reasoning, that characterize human cognition. This gap arises from a lack of informal learning, which thrives on interactive feedback rather than goal-oriented instruction. In this paper, we propose treating Games as a primary environment for LLM informal learning, leveraging their intrinsic reward signals and abstracted complexity to cultivate diverse competencies. To address the performance degradation observed in multi-task learning, we introduce a Nested Training Framework. Unlike naive task mixing optimizing an implicit "OR" objective, our framework employs sequential task composition to enforce an explicit "AND" objective, compelling the model to master multiple abilities simultaneously to achieve maximal rewards. Using GRPO-based reinforcement learning across Matrix Games, TicTacToe, and Who's the Spy games, we demonstrate that integrating game-based informal learning not only prevents task interference but also significantly bolsters the model's generalization across broad ability-oriented benchmarks. The framework and implementation are publicly available.

Figures (8)

• Figure 1: Overview of formal and informal learning paradigms, and a comparison between naive mixed training and the proposed nested training framework.
• Figure 2: Overview of the proposed nested training framework with formal and informal learning tasks. The middle panel presents a high-level abstraction of the nested task structure, while the right panel details the multi-turn nested training process with iterative model-environment interactions.
• Figure 3: Comparison between mixed and nested training in F + $\text{I}_2$ setting with 7B base model. Vertical axis denotes the performance, purple color denotes the ID tasks and gray color denotes the general abilities.
• Figure 4: Comparison between mixed and nested training in F + $\text{I}_2$ setting with 7B base model.
• Figure 5: Case Studies on MATH500 and CommonGen.