Towards Objective Metrics for Procedurally Generated Video Game Levels

TL;DR

Two simulation-based evaluation metrics that involve analysing the behaviour of an A* agent to measure the diversity and difficulty of generated levels in a general, game-independent manner are introduced.

Abstract

With increasing interest in procedural content generation by academia and game developers alike, it is vital that different approaches can be compared fairly. However, evaluating procedurally generated video game levels is often difficult, due to the lack of standardised, game-independent metrics. In this paper, we introduce two simulation-based evaluation metrics that involve analysing the behaviour of an A* agent to measure the diversity and difficulty of generated levels in a general, game-independent manner. Diversity is calculated by comparing action trajectories from different levels using the edit distance, and difficulty is measured as how much exploration and expansion of the A* search tree is necessary before the agent can solve the level. We demonstrate that our diversity metric is more robust to changes in level size and representation than current methods and additionally measures factors that directly affect playability, instead of focusing on visual information. The difficulty metric shows promise, as it correlates with existing estimates of difficulty in one of the tested domains, but it does face some challenges in the other domain. Finally, to promote reproducibility, we publicly release our evaluation framework.

Figures (7)

• Figure 1: Example levels for the Super Mario Bros. (left) and Maze (right) domains. In Super Mario Bros., the agent must traverse the screen to reach the flag at the rightmost edge, while in the Maze domain, the agent must find a path from the top left to the bottom right.
• Figure 2: Distributions of compression distance when using 3 different string representations of the Super Mario Bros. domain. This showcases a high sensitivity to an irrelevant factor---the specific representation used.
• Figure 3: Distributions of diversity metrics for the Maze domain. Each colour represents the metric values from pairwise comparisons over 5 seeds and 100 levels per seed for different level sizes. Only solvable levels that were generated using one specific method were considered (to fairly compare against the A* metric, which requires solvability), but the compression distance trend also holds for random levels.
• Figure 4: Illustrating the diversity metrics as level size increases in the Super Mario Bros. domain. (a--c) For all level representations, the compression distance displays sensitivity to the level size. Further, for a fixed size of levels, the different representations display drastically different distributions, mirroring \ref{['fig:cd_string_repr']}'s results. (d) Our diversity metric is robust to increases in level size.
• Figure 5: Example levels that are visually similar while playing identically. The solution path (shown in green) is identical for all levels.