Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analysis of Robustness of a Large Game Corpus

Mahsa Bazzaz, Seth Cooper

TL;DR

This work tackles the fragility of highly structured discrete game level data by introducing a formal data-robustness metric and a large, diverse corpus called the Generated Game Level Corpus (GGLC). The authors adapt robustness concepts to data, define discrete and continuous forms of non-robustness, and generate thousands of solvable and unsolvable levels across four tile-based games using a constraint-based generator (Sturgeon). Their analyses reveal substantial sensitivity to small input changes, with varying degrees across game types, and employ embedding-based methods (CLIP+UMAP) to compare robustness against standard benchmarks. The GGLC serves as a scalable resource to study PCGML under hard constraints, bridging communities working with structured data and providing a foundation for robust content generation research, with the dataset released under CC-BY 4.0.

Abstract

Procedural content generation via machine learning (PCGML) in games involves using machine learning techniques to create game content such as maps and levels. 2D tile-based game levels have consistently served as a standard dataset for PCGML because they are a simplified version of game levels while maintaining the specific constraints typical of games, such as being solvable. In this work, we highlight the unique characteristics of game levels, including their structured discrete data nature, the local and global constraints inherent in the games, and the sensitivity of the game levels to small changes in input. We define the robustness of data as a measure of sensitivity to small changes in input that cause a change in output, and we use this measure to analyze and compare these levels to state-of-the-art machine learning datasets, showcasing the subtle differences in their nature. We also constructed a large dataset from four games inspired by popular classic tile-based games that showcase these characteristics and address the challenge of sparse data in PCGML by providing a significantly larger dataset than those currently available.

Analysis of Robustness of a Large Game Corpus

TL;DR

This work tackles the fragility of highly structured discrete game level data by introducing a formal data-robustness metric and a large, diverse corpus called the Generated Game Level Corpus (GGLC). The authors adapt robustness concepts to data, define discrete and continuous forms of non-robustness, and generate thousands of solvable and unsolvable levels across four tile-based games using a constraint-based generator (Sturgeon). Their analyses reveal substantial sensitivity to small input changes, with varying degrees across game types, and employ embedding-based methods (CLIP+UMAP) to compare robustness against standard benchmarks. The GGLC serves as a scalable resource to study PCGML under hard constraints, bridging communities working with structured data and providing a foundation for robust content generation research, with the dataset released under CC-BY 4.0.

Abstract

Procedural content generation via machine learning (PCGML) in games involves using machine learning techniques to create game content such as maps and levels. 2D tile-based game levels have consistently served as a standard dataset for PCGML because they are a simplified version of game levels while maintaining the specific constraints typical of games, such as being solvable. In this work, we highlight the unique characteristics of game levels, including their structured discrete data nature, the local and global constraints inherent in the games, and the sensitivity of the game levels to small changes in input. We define the robustness of data as a measure of sensitivity to small changes in input that cause a change in output, and we use this measure to analyze and compare these levels to state-of-the-art machine learning datasets, showcasing the subtle differences in their nature. We also constructed a large dataset from four games inspired by popular classic tile-based games that showcase these characteristics and address the challenge of sparse data in PCGML by providing a significantly larger dataset than those currently available.

Paper Structure

This paper contains 14 sections, 3 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Structured Discrete Data
  4. Data in Games
  5. Methodology
  6. Domains
  7. Experiments
  8. Data Collection
  9. Measuring Non-robustness
  10. Results
  11. Distribution Non-robustness
  12. Sample Non-robustness
  13. Discussion
  14. Conclusion

Figures (5)

  • Figure 1: Solvable (blue) and unsolvable (red) levels can be only different by a single tile change or swap. Unacceptable levels in the upper row become the acceptable level in the lower row after changing or swapping a single tile.
  • Figure 2: Sample examples of violation of local constraints in different games. (From left to right: Cave), Platform, and Vertical
  • Figure 3: Example sample of each dataset. Cave levels come in different versions.
  • Figure 4: Level solution examples. Some solutions come in the format of the player's path in the level, and some (crates) come in the format of step by step playthrough.
  • Figure 5: UMAP visualization of CLIP embedding space with blue for solvable levels and red for unsolvable levels. The purplish color appears where the two classes, solvable and unsolvable, overlap.