G-PCGRL: Procedural Graph Data Generation via Reinforcement Learning

TL;DR

The G-PCGRL method is proposed, a novel and controllable method for the procedural generation of graph data using reinforcement learning that is capable of generating graph-based content quickly and reliably to support and inspire designers in the game creation process.

Abstract

Graph data structures offer a versatile and powerful means to model relationships and interconnections in various domains, promising substantial advantages in data representation, analysis, and visualization. In games, graph-based data structures are omnipresent and represent, for example, game economies, skill trees or complex, branching quest lines. With this paper, we propose G-PCGRL, a novel and controllable method for the procedural generation of graph data using reinforcement learning. Therefore, we frame this problem as manipulating a graph's adjacency matrix to fulfill a given set of constraints. Our method adapts and extends the Procedural Content Generation via Reinforcement Learning (PCGRL) framework and introduces new representations to frame the problem of graph data generation as a Markov decision process. We compare the performance of our method with the original PCGRL, the run time with a random search and evolutionary algorithm, and evaluate G-PCGRL on two graph data domains in games: game economies and skill trees. The results show that our method is capable of generating graph-based content quickly and reliably to support and inspire designers in the game creation process. In addition, trained models are controllable in terms of the type and number of nodes to be generated.

Figures (8)

• Figure 1: Graph generation with G-PCGRL: A G-PCGRL model is controllable through a given configuration to generate a valid graph from random noise according to a set of constraints on which it has been trained.
• Figure 2: Overview of the representation of a graph (right) as an extended adjacency matrix $M$ (left). The nodes are represented on the matrix's diagonal and are encoded depending on the node type. Edges between nodes are represented as 1, 0 for no connection.
• Figure 3: Overview of the action and observation spaces for the graph-narrow (left) and graph-wide (right) representation. The action space is highlighted with a green area, the observation space with red frames. The diagonal (blue) is static for both representations and can be configured at initialization or is generated randomly. Since we are only dealing with undirected graphs so far, the gray area is not used, but is important for defining the shape of a square matrix.
• Figure 4: An example set of constraints for graph data generation, consisting of four rules and three different node types. The same constraints are written with generalized symbols (left) and in a game economy use case (right). G-PCGRL can be trained with an arbitrary set of constraints, we will experiment with several ones that are listed in the appendix (Fig. \ref{['fig:app-constraints']}).
• Figure 5: The overall performance of the controllable G-PCGRL models compared to the different representations based on the maximum controllable graph size from a sample of 500 each. Performance is measured as the percentage of valid graphs generated (left). The average iterations per maximum graph size are shown in the right subplot. We compare both newly introduced representations to the original PCGRL-wide as a baseline (gray).