Application of predictive machine learning in pen & paper RPG game design
Jolanta Śliwa
TL;DR
The paper addresses automated monster-level estimation for Pathfinder Second Edition using ordinal regression, constructing a dedicated dataset and evaluating domain-aware metrics under chronologically ordered splits. It compares a broad spectrum of methods—from classical regression with rounding to specialized ordinal models and neural architectures—finding tree-based ordinal models (e.g., ORF, RF, LightGBM) to consistently offer the strongest performance across macro MAE/RMSE, Somers' D, and accuracy metrics. The study demonstrates that with appropriate evaluation design and threshold handling, ML models can reliably predict monster levels, enabling faster, more balanced encounter design in RPGs. This work provides a practical foundation for AI-assisted RPG design tools, with potential to automate balancing, generate monsters, and assist Game Masters in crafting engaging experiences while acknowledging temporal data dynamics in published content.
Abstract
In recent years, the pen and paper RPG market has experienced significant growth. As a result, companies are increasingly exploring the integration of AI technologies to enhance player experience and gain a competitive edge. One of the key challenges faced by publishers is designing new opponents and estimating their challenge level. Currently, there are no automated methods for determining a monster's level; the only approaches used are based on manual testing and expert evaluation. Although these manual methods can provide reasonably accurate estimates, they are time-consuming and resource-intensive. Level prediction can be approached using ordinal regression techniques. This thesis presents an overview and evaluation of state-of-the-art methods for this task. It also details the construction of a dedicated dataset for level estimation. Furthermore, a human-inspired model was developed to serve as a benchmark, allowing comparison between machine learning algorithms and the approach typically employed by pen and paper RPG publishers. In addition, a specialized evaluation procedure, grounded in domain knowledge, was designed to assess model performance and facilitate meaningful comparisons.