lmgame-Bench: How Good are LLMs at Playing Games?

TL;DR

lmgame-Bench reframes video games as robust, multi-turn benchmarks for state-of-the-art LLMs by introducing perception, memory, and reasoning scaffolds to stabilize evaluation and mitigate data contamination. It couples six diverse games with a Gym-style API and standardized prompts, enabling reliable discrimination among models and revealing that different games probe distinct capability blends. The work also shows that RL training on a single game can transfer to unseen games and planning tasks, suggesting games are effective for both evaluation and general-purpose skill shaping. Comprehensive analyses—including correlation, low-rank factorization, and ablations of harness components—highlight how games map to core capabilities (language, math/coding, visual/spatial reasoning, symbolic/puzzle solving) and identify practical limitations such as SMB randomness and computational costs.

Abstract

Playing video games requires perception, memory, and planning, exactly the faculties modern large language model (LLM) agents are expected to master. We study the major challenges in using popular video games to evaluate modern LLMs and find that directly dropping LLMs into games cannot make an effective evaluation, for three reasons -- brittle vision perception, prompt sensitivity, and potential data contamination. We introduce lmgame-Bench to turn games into reliable evaluations. lmgame-Bench features a suite of platformer, puzzle, and narrative games delivered through a unified Gym-style API and paired with lightweight perception and memory scaffolds, and is designed to stabilize prompt variance and remove contamination. Across 13 leading models, we show lmgame-Bench is challenging while still separating models well. Correlation analysis shows that every game probes a unique blend of capabilities often tested in isolation elsewhere. More interestingly, performing reinforcement learning on a single game from lmgame-Bench transfers both to unseen games and to external planning tasks. Our evaluation code is available at https://github.com/lmgame-org/GamingAgent/lmgame-bench.

Figures (17)

• Figure 1: lmgame-Bench uses modular harnesses—such as perception, memory, and reasoning modules—to systematically extend a model’s game-playing capabilities, allowing the model to engage with a simulated game environment through iterative interaction loops.
• Figure 2: (Left) Example from Ace Attorney showing contradictions in O3-generated text vs. ground truth. (Right) Effect of mitigation on similarity-performance correlation; red and blue lines show correlations with old and new leaderboard ranks, respectively.
• Figure 3: Spearman Correlation among lmgame-Bench and other benchmarks.
• Figure 4: Top‑weight benchmarks under each feature after low‑rank decomposition.
• Figure 6: Success rates during training across different evaluation tasks, with models trained on Sokoban, GSM8K, or a half-half mixture of both.