Grammar-based Game Description Generation using Large Language Models

TL;DR

The paper tackles generating grammatically valid game descriptions in Ludii GDL from natural language by introducing a grammar-guided two-stage framework. It first derives a minimal grammar $\\hat{G}[y]$ from the full GDL and then generates the description $\\hat{y}$ under that grammar, with iterative grammar and description refinement using an Earley parser. Empirical results show that grammar-based iterative decoding improves compilability, functionality, and semantic alignment over direct LLM generation, and that combining with SFT yields the best overall performance. The approach meaningfully narrows the gap between natural-language inputs and executable, syntactically correct GDL outputs, enabling more reliable automated game design and evaluation. Limitations include longer descriptions challenging the model and higher inference costs, suggesting avenues for model and prompt optimizations and human-in-the-loop validation.

Abstract

Game Description Language (GDL) provides a standardized way to express diverse games in a machine-readable format, enabling automated game simulation, and evaluation. While previous research has explored game description generation using search-based methods, generating GDL descriptions from natural language remains a challenging task. This paper presents a novel framework that leverages Large Language Models (LLMs) to generate grammatically accurate game descriptions from natural language. Our approach consists of two stages: first, we gradually generate a minimal grammar based on GDL specifications; second, we iteratively improve the game description through grammar-guided generation. Our framework employs a specialized parser that identifies valid subsequences and candidate symbols from LLM responses, enabling gradual refinement of the output to ensure grammatical correctness. Experimental results demonstrate that our iterative improvement approach significantly outperforms baseline methods that directly use LLM outputs. Our code is available at https://github.com/tsunehiko/ggdg

Figures (10)

• Figure 1: An example of Ludii game description for the game "Tic-Tac-Toe".$x$ is text that explains games in natural language. $y$ is a game description in Ludii GDL, a Game Description Language.
• Figure 2: An overview of our framework for grammar-based game description generation.We generate the game description $y$ from a natural language query $x$ using Large Language Models (LLMs). First, we generate the grammar $\hat{G}[y]$ required to construct $y$ in the Rule Decoding stage, and then generate $\hat{y}$ based on $\hat{G}[y]$ in the Game Description Decoding stage. The core of our framework is iteratively decoding by leveraging the grammar of game description languages (GDLs) to improve the initial response from LLMs. We make use of Ludii GDL as our GDL, which can model a larger variety of games and is a context-free grammar.
• Figure 3: Minimal Backus-Naur Form (BNF) grammar $G[y]$ for Tic-Tac-Toe. Redundant rules are omitted for simplicity.
• Figure 4: An example of grammar-based game description generation result. We generate game descriptions in two stages: first we generate the required grammar, and then generate the game description based on the grammar. The prompt includes demonstrations for in-context learning and the test example query $x$, and the demonstration contains the grammar $G[y^{(i)}]$. In the first stage, the minimal grammar $\hat{G}[y]$ that composes $y$ is generated. In the second stage, game description $\hat{y}$ is generated based on the generated $\hat{G}[y]$. The red arrow indicates that $\hat{G}[y]$ generated in the first stage is included in the prompt for the second stage.
• Figure 5: Processing flow of Rule Decoding stage. The Rule Decoding stage starts from the minimal grammar $\hat{G}[y]$ necessary to compose $y$ generated by the LLM, and improves it iteratively. From the grammar $\hat{G}[y]$, the set of rules included in the original grammar $G$ is extracted as $\hat{G}[y]_\mathrm{valid}$. Next, undefined non-terminal symbols $N_U$ are extracted from $\hat{G}[y]_\mathrm{valid}$. The rules $G_{N_U}$ concerning $N_U$ are obtained from the original grammar $G$ and input to the LLM along with $\hat{G}[y]_\mathrm{valid}$. The LLM then generates rules $\hat{G}{N_U}$ for the undefined non-terminal symbols. Finally, $\hat{G}[y]$ is updated by combining $G_{N_U}$ with $\hat{G}[y]_\mathrm{valid}$.