Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games

Yuchen Li, Cong Lin, Muhammad Umair Nasir, Philip Bontrager, Jialin Liu, Julian Togelius

TL;DR

GVGAI-LLM presents a reproducible, language-only benchmark for evaluating LLMs in symbolic, spatially grounded games by translating VGDL-based game states into natural language and using zero-shot prompting with interpretable metrics. The core approach couples a Translator (VGDL to text), a Language-based Player, and a strict per-step interaction loop, with metrics including Meaningful Step Ratio, Step Efficiency, Win Rate, and Normalized Reward to produce an Overall Score. Across 118 games, GPT-4o-family models show substantial limitations in spatial reasoning and long-horizon planning, despite token-heavy prompts, while some models display partial strengths in puzzle-like levels; RL baselines tend to underperform in planning-heavy tasks. The results highlight persistent gaps in symbolic understanding and propose prompt-design strategies (explicit coordinate tagging, verbose grounding) to mitigate spatial errors, positioning GVGAI-LLM as a valuable tool for diagnosing and advancing LLM agentic reasoning in structured decision problems. The work sets the stage for future work where LLMs could even design games, enabling a broader testing ground for rule-based reasoning and generative capabilities in symbolic environments.

Abstract

We introduce GVGAI-LLM, a video game benchmark for evaluating the reasoning and problem-solving capabilities of large language models (LLMs). Built on the General Video Game AI framework, it features a diverse collection of arcade-style games designed to test a model's ability to handle tasks that differ from most existing LLM benchmarks. The benchmark leverages a game description language that enables rapid creation of new games and levels, helping to prevent overfitting over time. Each game scene is represented by a compact set of ASCII characters, allowing for efficient processing by language models. GVGAI-LLM defines interpretable metrics, including the meaningful step ratio, step efficiency, and overall score, to assess model behavior. Through zero-shot evaluations across a broad set of games and levels with diverse challenges and skill depth, we reveal persistent limitations of LLMs in spatial reasoning and basic planning. Current models consistently exhibit spatial and logical errors, motivating structured prompting and spatial grounding techniques. While these interventions lead to partial improvements, the benchmark remains very far from solved. GVGAI-LLM provides a reproducible testbed for advancing research on language model capabilities, with a particular emphasis on agentic behavior and contextual reasoning.

GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games

TL;DR

GVGAI-LLM presents a reproducible, language-only benchmark for evaluating LLMs in symbolic, spatially grounded games by translating VGDL-based game states into natural language and using zero-shot prompting with interpretable metrics. The core approach couples a Translator (VGDL to text), a Language-based Player, and a strict per-step interaction loop, with metrics including Meaningful Step Ratio, Step Efficiency, Win Rate, and Normalized Reward to produce an Overall Score. Across 118 games, GPT-4o-family models show substantial limitations in spatial reasoning and long-horizon planning, despite token-heavy prompts, while some models display partial strengths in puzzle-like levels; RL baselines tend to underperform in planning-heavy tasks. The results highlight persistent gaps in symbolic understanding and propose prompt-design strategies (explicit coordinate tagging, verbose grounding) to mitigate spatial errors, positioning GVGAI-LLM as a valuable tool for diagnosing and advancing LLM agentic reasoning in structured decision problems. The work sets the stage for future work where LLMs could even design games, enabling a broader testing ground for rule-based reasoning and generative capabilities in symbolic environments.

Abstract

We introduce GVGAI-LLM, a video game benchmark for evaluating the reasoning and problem-solving capabilities of large language models (LLMs). Built on the General Video Game AI framework, it features a diverse collection of arcade-style games designed to test a model's ability to handle tasks that differ from most existing LLM benchmarks. The benchmark leverages a game description language that enables rapid creation of new games and levels, helping to prevent overfitting over time. Each game scene is represented by a compact set of ASCII characters, allowing for efficient processing by language models. GVGAI-LLM defines interpretable metrics, including the meaningful step ratio, step efficiency, and overall score, to assess model behavior. Through zero-shot evaluations across a broad set of games and levels with diverse challenges and skill depth, we reveal persistent limitations of LLMs in spatial reasoning and basic planning. Current models consistently exhibit spatial and logical errors, motivating structured prompting and spatial grounding techniques. While these interventions lead to partial improvements, the benchmark remains very far from solved. GVGAI-LLM provides a reproducible testbed for advancing research on language model capabilities, with a particular emphasis on agentic behavior and contextual reasoning.

Paper Structure

This paper contains 32 sections, 3 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Games as LLM Benchmarks.
  4. GVGAI Framework.
  5. Benchmark Design
  6. Environment and Agent Design
  7. Translator
  8. Player
  9. Prompt Configuration
  10. LLM Agent Interaction Pipeline
  11. Contextual Prompting
  12. Evaluation Metrics and Scoring Framework
  13. Meaningful Step Ratio
  14. Step Efficiency
  15. Win Rate
  16. ...and 17 more sections

Figures (4)

  • Figure 1: Overview of GVGAI-LLM
  • Figure 2: VGDL rule mapping and level layout mapping
  • Figure 3: Comprehensive score of each model across six GVGAI games. The comprehensive score aggregates meaningful action ratio, inverse steps, and (when available) reward and win rate.
  • Figure 4: Win rate comparison of three LLM agents across six GVGAI games. The win rate is computed as the percentage of successful completions out of multiple rollouts.